CA5 and/or CA6 could have very elliptical orbits and the planets apogee from it's star could be at roughly the same distance.
But planets aren't stationary. As I said, it's not just about the distance from the star, it's about all the parameters of the orbit. Even if, by some astonishing coincidence, Ceti Alpha V happened to be at the same point in space where Ceti Alpha VI was supposed to be by then, it would still have been moving along at least a somewhat different trajectory, and the ship would've had to match that trajectory in order to go into orbit, so they would've measured it and known that it wasn't the same.
Not to mention that they could count
. They could see the whole system -- from quite a distance away, given what's been shown about the power of long-range sensors in the Trek universe -- and know that it was missing one planet. Again, they should've been able to directly detect the explosion itself. Even if they somehow missed it, even if we're assuming that the visible light from the event wasn't detectable to them at warp, they certainly should've noticed the increased debris in the system -- that's something a starship would routinely scan for due to the potential navigational hazard.
And none of this addresses the question of why Starfleet didn't have a record of Khan's presence in this system. I have a hard time believing Kirk would've been so criminally irresponsible as to leave Starfleet unaware of the fact that there was a potentially dangerous population of superhuman criminals living in the system and that they should probably be left alone. Even if you posit some monumental bureaucratic snafu that kept the information from getting into Reliant
's databanks, the film itself posits that Chekov was aboard during the Botany Bay
incident -- so why doesn't he remember that this is the system where they dropped off Khan's people? For that matter, why doesn't Kyle? He was in "Space Seed." He was probably the guy who beamed them down to Ceti Alpha V. And he was on the Reliant
crew. The plot only works if two
people who were present for the first Khan incident completely forgot what planet Khan had been left on.