Essentially, the seeming defeat of Communism was also the seeming defeat of Enlightenment itself. Senseless worlds in fantasy express the inability to comprehend the real world. Plus, in fantasy it is much easier to write retreat into daydreams. What SF does keep its popularity is largely versions of superheroes and military SF, both of which are very much at the daydreaming pole. Fantasy isn't actually required to be mindless masturbation of various types, but the fantasy that isn't is neither common nor popular.