As I said, the Reliant's Star Charts were most likely inaccurate due to intervention by Kirk and Spock.Chekov and Terrell seem aware of the Ceti Alpha system and the number of planets in the system. They beamed down to what they think is Ceti Alpha VI and Chekov makes jokes about it's nature. You'd think sensors would have told them a planet was missing when they arrived. Especially if the planet is the sixth of six planets but there are only five.
I think the Reliant had a list of potential planets for use in the Genesis Project and Ceti Alpha VI was on the list. Probably based on previous explorations. Spock knew that Ceti Alpha V was habitable , if a bit savage in Space Seed, so it goes to reckon that there was information on the other planets as well.
Ceti ALpha V and its inhabitants were most likely removed from the official charts to avoid anybody going to look for it.
The system was likely picked in the first place, to hold Khan and his followers due to the fact that it held nothing of significant value and was off the beaten path.
Thus no one would have any reason to go there.
The Reliant may have just happened to pick the system because long range scans showed planets that might fit Dr. Marcus' requirements of no life forms.
It was just extremely bad luck and a massive coincidence that Khan was there.
(which, I might add, is a Star Trek staple)