To be sure, we have only the vaguest idea of how the laws of physics govern what happens in faraway solar systems. Most of our original assumptions, reasonable at the time, have been proven false when we have been able to actually observe solar systems other than our own. Some have been blown out of the water by more detailed study of our own system, too - say, the concept that the Sol system would have to be classically stable simply because it has remained in existence for a few billion years. Between Newton's stability and Velikovsky or Molchanov's random chaos, it seems that current science agrees that at least the inner Sol system is unstable enough to allow for Mercury to start careening wildly out of orbit any minute now.
Whether that could happen at Ceti Alpha is impossible to tell even if we knew the exact orbits, masses and so forth of the planets there - an at least seven-body problem should already be chaotic enough to allow for basically anything to transpire.
Moreover, Khan would have had an even vaguer idea of what was going on. His instrumentation would be lacking, as no doubt would his knowledge on astronomy. He said CA6 "exploded", but for all we know, he just saw it there one night and didn't see it any more the next night, and blamed the later climate changes on that disappearance while inventing all sorts of additional details to complete the theory.
We don't really have to assume all that big a cataclysm in order to explain the data: the slightest shift in CA5's orbit or orientation could explain the climate change, and indeed a major shift would probably have terminated the Class M environment and the penal settlement there at once.
The issue of how the Reliant came to mistake CA5 for CA6 is actually quite separate from what really happened in the system. We have every reason to assume that the CA system was poorly charted to begin with, hence ideal for marooning a few dozen supermen. We have little reason to assume that the Reliant would have had available means for identification more refined than just flying in, counting how many planets there were, and assigning ordinals per distance from the star, then comparing those ordinals with whatever was in the databases.
And we know that what was in the databases was vague to begin with. The Reliant crew probably knew there would be a desertlike Class M planet there, or else they wouldn't have come looking for one (unless they were doing a random search within a specific area). But they didn't know any details of that planet, because they did have to come there and have a look before they could decide whether it would fit the Genesis test parameters. It wouldn't have occurred to them to painstakingly study the orbits of the planets before homing in on the obvious desert world and identifying it as CA6. And that wasn't their job, either.
So perhaps CA6 exploded, perhaps it didn't. Perhaps its demise affected CA5 orbit and climate, perhaps something else did. I'm rather inclined to think that CA6 fell victim to something "unnatural" yet fairly mundane in the Trek universe, such as the impact of a giant derelict starship or an out-of-control doomsday weapon, the final shutdown of a divine machine that held the planet together, or a misfire of the Caretaker's interstellar abduction beam. But I think it would be quite reasonable to accept the possibility of a random instability within the known laws of nature, too.
Timo Saloniemi