Current party size (4) is a tad bit small. When you get some cool people who aren't in your party, but you still team up with them things can get confusing.
Since there are no rules against teaming, I don't think there should be a cap to the total party size at all. While obviously, forming a party makes things a lot easier (like identifying friend from foe + group chat), having it capped will and has already lead to some players just being excluded from the fun, especially if they die.
There's two ways to handle this: Either increase cap to a much higher amount (5-9) or add a 'leader' perk, that allows the party leader to invite more members the higher the perk is upgraded.
A downside to this suggestion is that the enhanced communication allows well coordinated parties to become extremely powerful. Although if someone is able to succesfully lead a well coordinated party, I believe it should be rewarded.
If this is added, I think it would encourage teamplay further and this could lead to more dedicated players - after all, players who stay together have more fun and are more likely to return to the server to play again.