And I'm not saying it is. I'm saying that's not an excuse to abandon it altogether. It's not even about science. Researching science for an SF story is no different from researching the geography or culture of the city where your story is set, or researching theater culture if you're writing a story about actors, or consulting with a chef if you want to write a story about a restaurant. There's this widespread idea that if you write something pertaining to science or science fiction, that's somehow an excuse to abandon the basic standards of competent research that would apply when writing about any other subject, and that just doesn't make sense. I don't buy the excuse that most readers won't know the difference, because most readers won't know the difference if you don't get a recipe right in the restaurant story or if you don't get trial procedure right in a courtroom drama. But some people in the audience will notice the difference, and they will complain, and I think it's unfair to them to think their reactions don't matter just because they're in the minority. No matter what the subject, it's worth doing your due diligence. Yes, of course the story is the starting point, but part of serving the needs of that story is to build a plausible setting for it to take place in.
(And just for the record, sometimes the science is the starting point. A number of my story ideas have come to me because I looked at a scientific or technological question and asked, "What are the consequences of this?" For instance, my first published story, "Aggravated Vehicular Genocide," was inspired by the idea of an interstellar ramjet starship and the powerful lasers it would need to vaporize or deflect asteroids in its path. One day it occurred to me to wonder, "What if an alien habitat crossed its path instead?" Thinking through the science gave me the story.)