Maybe I'm being influenced by McIntyre's novel, but the reasonable explanation (such as it were) is that the Ceti Alpha (Alpha Ceti) system is nowhere. It's like someone complaining why didn't they know where the rocks were in the middle of Nevada in the 1920s.
Doesn't work, though. The
Reliant had prior knowledge of the existence of at least six planets in the system, which means the system has been surveyed, either telescopically or by the
Enterprise when it dropped off the Augments. If they knew the planet existed, if they knew it was likely to be lifeless, then they would've known its orbital and physical parameters and would not have been so easily fooled. Plus, again, the energy required to destroy a terrestrial planet would be equal to the Sun's entire energy output for a week, so the event would've been bright enough to outshine the star itself for a time. The
Reliant would've seen the explosion 15 (18) light-years away when they passed through the wavefront of the light, if Federation subspace telescopes hadn't seen it when it happened 15 (18) years earlier. They couldn't have
not known.
Earthbound writers often fail to realize that there are no horizons in space. We can see galaxies billions of light years away. With a good enough telescope, you can see basically everywhere in the universe, except for things that are obscured behind dust clouds or nebulae or the mass of the galaxy (which is why we know less about its far side than we know about distant galaxies). But most any star system close enough to be reached by a starship (barring wormholes, exotic drives, and transgalactic abductions) is close enough to have been mapped telescopically beforehand. And of course said starship could do its own telescopic survey as it drew near.
The aim of the story is to accidentally happen upon a very pissed off Khan Singh. Who was never reported because Meyer thinks Kirk is a shoot from the hip cowboy. It's all part of Meyer's "Kirk was a reckless young legend and now he will pay the price."
None of which makes sense. Kirk was a student of history. Discovering 70-odd living people from a period of Earth history for which records were incomplete, including at least one infamous world leader, would've been a gold mine for historians. There's no way he wouldn't have reported it.
In my movie-era novels, I've implied that Starfleet Intelligence and/or Section 31 classified Kirk's report because the Klingons see human Augment research as a military threat, because of what happened in "Affliction"/"Divergence" a century earlier. So Kirk did let them know, but they buried it for fear of provoking the Klingons.
Still, there must surely have been a better way for the movie to set up the escape of Khan's people. If Ceti Alpha V had been established as a proper penal colony for them, with historians coming in to interview and study them, Khan could've manipulated that to arrange an escape. Then he somehow finds out about Genesis and goes after it, or maybe the MacGuffin is something other than Genesis and we avoid the nonsense of that technology -- although that would make it harder to resurrect Spock after Nimoy changed his mind about leaving.