I think the problem with the way it’s presented in Wrath of Khan is that going into the Ceti Alpha system is preceded by Carolyn Marcus being adamant that Starfleet be sure that they’ve done due diligence in surveying where they’re going to use the Genesis Device.My point is did anyone feel it worth observing compared to the billions of other stars?
Nobody bothering to keep an eye on Ceti Alpha doesn't necessarily mean Incompetence. A limited number of astronomers requires them to be selective.
Why should they keep an eye on Ceti Alpha? The fact 72 criminals who attempted to hijack the Enterprise were exiled there simply may not be a good enough reason.
So you’re left with a bunch of options:
1) The crew of the Reliant, even though they assured Marcus that they would be thorough, were incompetent and didn’t know how to access survey records for the system. Which I don’t feel is the case, since nothing else in the story colors Chekhov, Terrell, or the rest of the Reliant crew to be inadequate. If they were being lazy about it or blowing off Marcus’s requirements, I don’t think they beam down to Ceti Alpha V in the first place to check out the anomaly in the scanner.
2) Either intentionally or through a glitch, the records of the events of “Space Seed” aren’t available to the Reliant crew. One interesting aspect about this in the movie is that Chekhov is aware of Khan and knows he was left on Ceti Alpha V, which Khan realizes it means Chekhov didn’t expect to find him since they didn’t know they were on Ceti Alpha V. But Captain Terrell has to have Khan’s existence explained to him, like he’s hearing about it for the first time. That would imply that Chekhov KNEW they were in the same system but didn’t inform the captain that you know 20 years ago we left a 20th century dictator and his crew of genetic supermen in this system. So why would Chekhov not disclose that info from jump, or why wouldn’t the computer give that info to the Reliant crew?
If you go back and watch the ending of “Space Seed,” yes there’s a hearing. But Kirk shuts it down almost immediately and drops all of the charges. The entire tone of Kirk’s “solution” is trying to find a way where Khan and McGivers aren’t arrested and turned into criminals.
Captain's Log. Stardate 3143.3. Control of the Enterprise has been regained. I wish my next decisions were no more difficult. Khan and his people. What a waste to put them in a reorientation centre. And what do I do about McGivers?
So that’s why the idea that maybe Kirk sidestepped putting all of this down in a report becomes more believable. It would also explain why, as Khan complains, no one came around for decades to check on their colony, and the Reliant’s survey of the system didn’t set off any alarms when it didn’t match with the Enterprise’s.