Think of it the following way: when you drive to your house, do you recognize it as your house or as the fourth house on the left? The explosion that destroyed Ceti Alpha VI laid the fifth planet to waste and changed its orbit, making it look like the sixth planet in the process. Add to that the fact that the planets aren't arranged in a straight line (each has a unique orbit), and it's not hard to see why Reliant didn't realize they'd stopped at the fifth planet instead of the sixth.
But that's ridiculous. There are no horizons in space, no obstacles. You can see everything. I mean, look at us -- we're currently discovering tons of planets in other star systems dozens or even hundreds of light-years away, and continuing to discover new dwarf planets and minor objects way out on the fringes of our own system. Not only would it have been easy for the Reliant
's sensors to detect every
planet in the system, not to mention any debris left by an exploding one, but they should've detected the actual explosion itself
as soon as they came within about 15 light-years of the system, because that's when they would've intersected the light given off by the event 15 years earlier (or 18 years, if you go by the Okuda chronology, in which case they would've detected it even further away).
Not to mention that there's no way the explosion of a planet would've caused the next planet inward to somehow get pulled outward
in its orbit and exactly duplicate the orbital parameters of the exploded planet -- not only orbital radius, but eccentricity, inclination, and argument of perihelion, or at least close enough that it would be exactly where Starfleet records said it would be a full 15 (or 18) years later. We're not talking about billiard balls on a table. More like billiard balls spread out over the Mojave Desert, and in constant motion. It's hard to find them unless you know where to expect them to be at a given time -- and the odds of one happening to end up in the same place as another are minuscule.
Of course, this is far from the only plot or conceptual flaw in TWOK. The movie is full to the brim with flaws. How come Khan's followers, who were multiethnic adults when they were stranded 15 (or 18) years earlier, are now a bunch of blond Nazi-recruiting-poster types in their 20s? How can a tiny torpedo terraform an entire planet? How can a torpedo programmed to restructure the surface of an existing planet spontaneously adapt its programming to create a planet (and implicitly even a star) out of nebular gas? How can the supposedly ingenious Khan fail to see through Spock's screamingly obvious "hours could seem like days" code? Why did the Ceti eel just leave Chekov instead of killing him, which according to Khan was the inevitable result? Why does Scotty bring his bloody, dying cadet/nephew (depending on the cut) to the bridge instead of sickbay? And how in the seven hells of Mongo can Kirk say he's "never faced death" after losing Gary Mitchell, Edith Keeler, Sam and Aurelan, Miramanee, and his unborn child??????