I think there is no single reason for Star Treks demise, but a number of them:
- TNG started weak and got better throughout it's run the same happened with DS9. I think the later seasons of DS9 were the peak of televised Star Trek. From there it all went downhill
- VOY was probably the biggest nail in ST's coffin. The writing was sloppy, the plot all over the place and Janeway was just a stark-raving nutter, not to mention 7 O'9 as a shameless titillation tool
- Then came the prequel idea of ENT. Good idea, but god awful realization. The first two seasons were all over the place with rare gems like "The Andorian Incident" or "Cease Fire" (practically every episode with Shran in them), god awful episodes like ANIS or unimaginative rehashes of concepts, like "Oasis". The early seasons had so many plot-holes, the ride rattled the fillings out of my teeth.
- And the final nail in the coffin was TATV. Not only was is biblically bad, it was as if B&B just took the piss out of the fans. They practically killed the franchise by planting a fat ol' reset button on the whole ENT universe, as we all know now it was all only a holodeck fantasy of Fat!Riker