Its the Federation would not some of the member worlds allow for group marriages?
It's already canonically established that the Federation allows for more marriages than the heterosexist monogamy we associate with marriage in the present-day United States. We know that Bolians, for instance, have group marriages -- "Field of Fire" (DS9) establishes the existed of Bolian families that feature a wife and co-husbands.
We also know from TNG's "Data's Day" that Andorian marriages require groups of four; the novels have run with an interpretation of this, establishing that Andorians have four sexes and that their "marriages" (more properly translated as bondgroups) require all four sexes before they are willing to reproduce.
And as
T'Girl already noted, Denobulans seem to practice an extended form of polygamy; granted, it's never been canonically established that Denobula joined the Federation, but it seems improbable to think that they did not, given their proximity to and good relationships with Earth and Vulcan.
The non-canonical Trek novels have also established quite clearly that same-sex marriages are legal and accepted on Earth by the pre-Federation ENT era, and that this continued into the 24th Century, even among alien cultures such as the Trill.
I would imagine that marriage laws, or the equivalents thereof, are a matter the Federation government mostly leaves to the Member governments to regulate.