You may be overstating how optimistic people were back in the old days. Every generation thinks that previous eras were happier than today, but in fact, those eras also thought that the world was going to hell in a hand basket.
And when it comes to science fiction, TREK was unusual in its day because it purported that there would be a bright and shiny future. Most SF shows and movies of the era were of a paranoid bent, full of alien invasions, post-atomic wastelands, soulless computers, etc. See INVASION OF THE BODY SNATCHERS, THE TWILIGHT ZONE, THE OUTER LIMITS, THE INVADERS, LAND OF THE GIANTS, PLANET OF THE APES, SOYLENT GREEN, FAHRENHEIT 451, DR. STRANGELOVE, THE OMEGA MAN, LOGAN'S RUN, MAD MAX, ESCAPE FROM NEW YORK, THE TERMINATOR, THE X-FILES, etc.
Dystopia were the norm, not the exception, long before 9/11. And even STAR TREK, back in the 1960s, acknowledged the Cold War paranoia of the time. Remember Roberta Lincoln talking about her generation was rebelling because they didn't know whether there was going to be a future, what with the Bomb and all?
Post-War America was not all sunshine and optimism. It was also backyard bomb shelters, the Cuban Missile Crisis, McCarthyism, Viet Nam, race riots, political assassinations, and the constant shadow of thermonuclear war. Heck, for a while there, pretty much every movie set in the "near-future" of the 1990s assumed the nineties were going to be a dark ages of barbarism and destruction. (Hi, Eugenic Wars.)
