Indeed, there is big difference between studying an alien civilization via duck-blind missions and spying on them in order to gain military intelligence.
There's a clear difference of intent, but that doesn't mean the actions are any different. Nor does it mean the subjects of such surveillance would perceive any such distinction if the surveillance were to be uncovered, because, really, how the hell do they know what the Federation intends to do with the information it gathers?
As for the idea that "we" (which I presume means, white folks) have destroyed every less-advanced civilization we've encountered:
Well, no. Just a majority of them. But apart from the spread of horrific depopulating diseases as an inadvertent result of contact with European diseases, Europeans didn't destroy those civilizations by accident. European cultures set upon the world with a deliberate imperialistic intent. The English decided to wipe out the nations of Central North America. The Spanish set out to conquer and enslave the natives of South America. The French and the Germans and the British all set out to conquer and oppress India and Africa. These were deliberate choices.
By contrast, Europe itself had been a backwater peninsula that was hopelessly behind the technological developments of the Islamic world and of Asia for many centuries when its contacts with those cultures led to European nations acquiring their technologies. Europeans were then able to build upon those technologies and then launch their own empires. So for Europe's case, contact with more technologically advanced cultures was beneficial.
So it really all depends on your intent. Contact with a less technologically advanced species won't automatically lead to oppression -- if you're careful not to be doing any oppressing. If the Prime Directive has a purpose, then, it's essentially about saying, "Listen, we ourselves are not morally advanced enough yet as a culture to be trustworthy in making contact with less technologically advanced cultures without dominating and oppressing them. This isn't about them being inferior to us, this is about us not being mature enough yet."