The Vietnam analogy is a silly one to make. Every war is unique.
But if we want to ask if the Federation has ever fought an unpopular war and lost(or at the least did not win) said war, then it becomes a more interesting question.
We have very rarely seen Federation civilians during times of war, and I can't really recall anything aside from the DS9 "Paradise Lost" stuff that dealt with it directly. But that is meaningless because it was all evil shapeshifters and conspiracy stuff.
Has any Federation civilian expressed an anti-war sentiment on-screen? Not since David Marcus, and even that was just a general loathing of the military, not a specific war.
So moving on since Federation public opinion has not really been depicted. Which wars has the Federation not won?
None really, they've had partial victory in every war, and more often total victory.
So no, I don't think Starfleet has ever had a "Vietnam" type experience.