Except for being a scientist or running for president, wouldn't becoming captain of a starship be the ultimate achievement in professional life in the 24th century?
Hell, no! Why would it? Command is just one career specialization. A captain would be useless without the rest of the crew and the expertise they provide.
I mean, think about it. A military has a pyramidal command structure. The higher the rank, the fewer people there are who hold it. Most people within a naval organization will never reach captain's rank or even commander's rank. Hell, most will never even become officers. Most career military personnel level out at lower ranks. Many will just serve in the military for a few years, then rotate out and find a career in civilian life.
Admiralty is a demotion from starship life to paper-work job (because Starfleet can't simply have an admiral be CO of a starship for some reason).
That's how James T. Kirk and Jean-Luc Picard thought about it. It doesn't follow that every single person in the Federation shares their biases. We know that Ben Sisko aspired to the admiralty rather than starship captaincy. Kathryn Janeway seemed to settle in nicely as an admiral. Jonathan Archer had a career path from captain to admiral to ambassador to councilor to president. So if anything, only 40 percent of Trek series leads have been shown to prefer ship captaincy to flag rank or other career paths.
Sure, personal fulfillment is possible also when one is a lieutenant or so, but the crown of professional achievement in the Federation would be captaincy. The analogy for today's world is a university degree, the preeminent civilian achievement.
That's not a good analogy at all. Captaincy is more like becoming the head professor of a specific university department. Any officer's rank would be the equivalent of a university degree. After all, again, command is merely one of the many specializations within Starfleet. Choosing the command path over sciences or engineering or security is like going for a management degree as opposed to physics or English or history.
And there are civilians in the Federation. It's important not to confuse the selection bias of the shows we see for an objective portrayal of the UFP as a whole. Starfleet is all-important to the characters we see because we're only seeing shows that are about Starfleet officers, people who are there because they thought that way. Starfleet may be the most prestigious research institution in the Federation because it's responsible for charting the frontiers of space, but it's not the only such institution. We know of numerous canonically established civilian institutions like the Vulcan Science Academy, the Daystrom Institute, the Cochrane Institute, the University of Alpha Centauri, the Regulus III Science Academy, etc.
There's also the matter that if one lives long enough one ends up being promoted every couple of years, eventually reaching captaincy by necessity.
No, that is absolutely not how it works in any actual military organization. It doesn't make any sense. After all, any single officer commands a large number of subordinates. The higher your rank, the larger the number of people below your rank, under your command. If an average starship crew is, say, 300 people, with only one captain per ship, that means that at most a third of a percent of Starfleet personnel will ever earn captain's rank.
People don't just get promoted because they've lasted long enough. It's also a function of whether there are slots available for promotion. That's why Harry Kim was an ensign for 7 years -- because there was no upward mobility on a ship where none of the crew ever rotated out to other posts.
Riker was offered captaincy repeatedly, and Vaughn remained commander to keep a low-profile. See Gomez and Ro, starting out as lowly ensigns in TNG and now four-pipped.
And Tuvok remained a lieutenant for decades. And O'Brien was a petty officer for decades. There's no single lockstep career path. Only some personnel are on the command track to begin with, and only some of those perform well enough to earn promotion to captain's rank. It's a highly competitive field, so of course most of the contenders will never reach their goal. Again, though, as television viewers we're subject to a selection bias, because we focus on the characters who are capable and gifted enough to earn promotions, rather than the majority of officers who perform competently but never break out of the pack.