I think we can't assume that an interstellar federation would function the same way as any government confined to a single planet. It would just be too much vaster in size and distance. There would have to be a greater degree of autonomy for each individual world, simply because it would be impossible for any single body to regulate hundreds of entire planets at once. So no, the Federation isn't like the UN, but it isn't like the United States either. It's something else.
The way I interpret it is that individual UFP worlds have a lot of autonomy over their internal affairs -- that as long as they follow certain basic precepts like having a unified global government, guaranteeing equal rights for everyone, and practicing an enlightened code of justice, they're free to make their own choices about how they manage their own local affairs. The Federation Council is responsible for interstellar affairs, matters that involve more than one world or that involve interactions with powers outside the UFP. That division of responsibility is necessary because if the Council had to deal with the internal affairs of hundreds of entire planets, it would be too overloaded to function.
So the reason the Klaestron wanted the Federation to stay out of their business is because it was an internal matter, the prosecution of a crime committed on their territory. Even if they were a UFP member, they would still have jurisdiction over their own internal legal affairs. One could argue that since Dax was from another member world, that made it an interstate matter falling under federal jurisdiction, but it's hardly unheard of for a state and a federal government to disagree over who has jurisdiction over a particular matter.