I've seen people getting taken out of Star Wars: The Last Jedi because 'people can't survive in space', and that's a story about space wizards, so the show about scientists learning about the universe has to try even harder. And I'm sure everyone has a point where a story will lose them, even people who reckon they've moved beyond childish things like caring about believability in their made-up TV show, so writers shouldn't be testing how far they can lower the bar.
TBH, I think a lot of it comes down to the tone of the fictional story, not how rigorous it is.
As an example outside of sci-fi, consider if you're reading or watching historical fiction which is supposed to take place in say the Victorian era, but the dialogue contains a lot of anachronistic modern slang. If the the subject is dark and serious (say covering labor strife, or a murder investigation) this will be jarring as hell. On the other hand, if it's a light or comedic story, you can get away with it.
The same goes in worldbuilding in speculative fiction. If you're telling a serious story, the world needs to be framed in a serious, rigorous manner. Not necessarily scientifically accurate, but consistent and grounded.
A great example of how tone and worldbuilding don't work well is the Netflix movie Bright. The concept of a gritty cop story told in a fantasy world had a lot of promise (has been done well in book form several times), but the worldbuilding was just a lazy pasting of fantasy races onto a bog-standard cop movie. Worse, the tone of the movie was deadly serious, without a hint of tongue in cheek. It ended up a drag and nonsensical, a world where a "dark lord" nearly took over 1,000 years ago, but somehow a Chicano character also referenced The Alamo.