It might be a good question to ask the OP why they think time travel is the more critical determination, or less believable, than each of the other elements of science fiction listed by various posts here.
It may come down to the fact that we know, basically, how certain things work.
Transporter: E=MC2.
Faster than light travel, she ships never actually reach the speed of light, but arrive at the destination as fast or faster than light by so called dune effect. That's where the warp engines come into play, the warp bubble around the ship shrinks the space in front of the ship and enlarges it behind it.
Even if these things are not possible, at least there's a theory behind them.
For time travel, there is no "explanation", it just happens... And time traveling must be pretty precise, one second in the wrong direction and who knows what might happen.
But why do we have to believe that this stuff is "possible" to enjoy it? It's fiction, an exercise in imagination. An entertainment.
I sincerely doubt that Jack the Ripper was actually an ageless energy being, but I'm not going to reject "Wolf in the Fold" because of it . . .

And I can think of lots of episodes that don't involve time-travel that aren't nearly as enjoyable as, say, "Tomorrow is Yesterday" or "Little Green Men."