That's a lot of questions.Think of it the following way: when you drive to your house, do you recognize it as your house or as the fourth house on the left? The explosion that destroyed Ceti Alpha VI laid the fifth planet to waste and changed its orbit, making it look like the sixth planet in the process. Add to that the fact that the planets aren't arranged in a straight line (each has a unique orbit), and it's not hard to see why Reliant didn't realize they'd stopped at the fifth planet instead of the sixth.
But that's ridiculous. There are no horizons in space, no obstacles. You can see everything. I mean, look at us -- we're currently discovering tons of planets in other star systems dozens or even hundreds of light-years away, and continuing to discover new dwarf planets and minor objects way out on the fringes of our own system. Not only would it have been easy for the Reliant's sensors to detect every planet in the system, not to mention any debris left by an exploding one, but they should've detected the actual explosion itself as soon as they came within about 15 light-years of the system, because that's when they would've intersected the light given off by the event 15 years earlier (or 18 years, if you go by the Okuda chronology, in which case they would've detected it even further away).
Not to mention that there's no way the explosion of a planet would've caused the next planet inward to somehow get pulled outward in its orbit and exactly duplicate the orbital parameters of the exploded planet -- not only orbital radius, but eccentricity, inclination, and argument of perihelion, or at least close enough that it would be exactly where Starfleet records said it would be a full 15 (or 18) years later. We're not talking about billiard balls on a table. More like billiard balls spread out over the Mojave Desert, and in constant motion. It's hard to find them unless you know where to expect them to be at a given time -- and the odds of one happening to end up in the same place as another are minuscule.
Of course, this is far from the only plot or conceptual flaw in TWOK. The movie is full to the brim with flaws. How come Khan's followers, who were multiethnic adults when they were stranded 15 (or 18) years earlier, are now a bunch of blond Nazi-recruiting-poster types in their 20s? How can a tiny torpedo terraform an entire planet? How can a torpedo programmed to restructure the surface of an existing planet spontaneously adapt its programming to create a planet (and implicitly even a star) out of nebular gas? How can the supposedly ingenious Khan fail to see through Spock's screamingly obvious "hours could seem like days" code? Why did the Ceti eel just leave Chekov instead of killing him, which according to Khan was the inevitable result? Why does Scotty bring his bloody, dying cadet/nephew (depending on the cut) to the bridge instead of sickbay? And how in the seven hells of Mongo can Kirk say he's "never faced death" after losing Gary Mitchell, Edith Keeler, Sam and Aurelan, Miramanee, and his unborn child??????

I wouldn't dare question your knowledge because your job to know everything about how the Trek universe works.
But (you saw that coming, right?) ...
The Ceti Alpha system was apparently remote maybe not well charted. Wasn't that why Kirk left Khan and Co. there? I don't spend a lot of time dissecting these things, but it makes sense to me. I'd think that the Khan incident was not "required reading" so I'm not surprised that the Reliant captain wasn't familiar with it. Add to that, it was 15 years earlier so it probably wasn't at the front of Chekov's mind. Not until he saw the "Botany Bay" thing anyway.
As I said in an earlier post, if CA5 was near where CA6 was supposed to be, why look any further? Planets don't just line up for roll call when a ship approaches, so if there's a planet where you're looking you might assume that it's the right one.
I can't address costume or makeup design or casting, because I don't know enough about it.
The Genesis device, as we found out later, used protomatter (whatever that is) for the terraforming process. We don't know what scientific advancements will be made in the next 300 years, so I just accept that as a neccessary suspension of disbelief.
Spock's code: maybe Khan was preoccupied or gloating or who knows what, but he didn't get it.
Chekov was apparently strong enough to resist the eel's influence and "sour the milk." Why? No idea. But it's there.
Why did Scotty haul his dead nephew to the bridge? Dramatic effect. No other reason I can see.
Kirk "never faced death" himself. Sure, he's seen it plenty but it was always someone else. He had never faced a situation that he couldn't BS his way out of. I'm pretty sure that's what he means.
I probably missed a few, but these are what I'm thinking after seeing the movie dozens of times over the last 30 years.
Again, I have a ton of respect for what you do and I'd like to know how you might have written it differently.