That's the point: one's oppression is another's irritation. And nobody in the UFP or Starfleet seems to be particularly interested in rooting out irritation as a thing. Or oppression as a thing. They just aim for a specific level of discomfort, which excludes killing the carpenters who try their hand at oil painting, but includes throwing into jail those who speak their mind in front of a superior officer.
It's just that the UFP comfort zone is not a particularly narrow one. Duels to the death over possession of females are okay. Whether a benign caste system could come up with something less repugnant than that isn't much to ask...
Really, it would be easy to see some people witness the UFP go to war over caste systems, and deem them savages for such senseless and immoral behavior. Overreacting to irritation is just as bad as underreacting to oppression.