The Federation definitely had its own self-interests that motivated its actions and priorities. DS9 was good at exploring that via the outside perspective.
Indeed, The Old Mixer. That's what I said that the UFP's expansion is at odds with its ideals (the Prime Directive, specifically).
As for my 'assumption' - it's not an assumption, it's what the characters of the show repeatedly said and how they acted. The 24th century Prime Directive IS an end unto itself.
See for example Dear Doctor, Homeward or even episodes where the crew made the sane decision in order to hear the pro prime directive arguments: Pen Pals, Who watches the watchers. For warp capable species, see for example Redemption or The circle.
The summary - the 'cosmic plan' is the best option for all species' evolution; any interference will corrupt, dirty this cosmic plan, leading to a less desirable evolutionary path, regardless of the circumstances.
This 'ridiculous extreme' IS what the 24th century Prime Directive has become (as per the intent of the writers).
But it wasn't taken to that extreme, and it wasn't a goal unto itself. One can be forgiven for thinking otherwise when the lead characters could get so preachy about it, but to suggest that the Prime Directive is or should be more important to the Federation than the drive to explore and expand is taking things to an extreme that wasn't demonstrated on the show. That would be the tail wagging the dog.
The bottom line of the PD is that it was a self-imposed rule that kept our heroes from taking the easy way out in many a situation. Their greatest limitations weren't defined by what they could
do, but what they would