I mean a four sexed species would be far fetched from an evolutionary standpoint but Star Trek has has given Barclay a spider for an ancestor and humanoid species frequently evolve into omnipotent non corporeal energy beings so I really don't think its too far out there for Trek.
Your memory fails you. Denobulans had two sexes only, but standard practice was for each husband to have three wives, and each wife to have three husbands, which made for complex marital relationships. Perhaps as a result of this, sexual fidelity did not seem to be among Denobulan ideals.