The Genesis torpedo was detonated in the middle of a nebula with not a planet in sight. Thus Genesis, created a planet from the material in the nebula. Genesis was never meant to do that.
Umm, sure there was a planet in sight when the Genesis device was detonated. As a matter of fact, the Enterprise had just departed the spherical, high-gravity asteroid Regula at a brisk walking pace a few minutes earlier. Odds are, this is exactly what the Genesis device transformed into the Genesis planet.
After all, as said, Genesis was never meant to transform a nebula. And Genesis wasn't supposed to be a flexible technology: as the Marcuses said, not a single byte more could be crammed into the programming (or some other aspect of the device that uses bytes?), so there probably wasn't an extra subroutine there for doing unexpected things.
So the Genesis detonation and the Genesis planet destruction would also conveniently erase the evidence from the original Genesis cave experiment...
It's also possible that there were other planets in the system for the device to act on. It would depend on our interpretation of the terminology. It seems that "Regula One" (as Kirk and Uhura both say) was the name of the laboratory, and "Regula" (as Spock once says) was the name of the asteroid. Systematically, we could argue that Regula was the star, and the star's planets and other solid satellites would be labeled Regula I through ??? - so if the 'roid is just plain "Regula" even according to the anal-retentively accurate Spock, then there might not exist a Regula II. But then again, we don't have to assume that the naming was systematic, not even when this seemed to be an insignificant system where nobody would have a specific interest for inventing "proper names" and other fancy exceptions.
Timo Saloniemi
The astronomical science of TWOK is shaky at best, but does allow for several different interpretations. Genesis may or may not have been formed directly from the Mutara Nebula, and could in fact have been a transformed Regula-I, as
Timo suggests, but the transformation of the nebula itself cannot be ignored.
I suspect that there probably
was a parent star named "Regula," while the planetoid was called Regula-I, leaving the station with the designation Space Lab-Regula I, as it was mostly referred to throughout the film especially during hailing communications. Spock's more relaxed statement, "Regula is Class-D" simply may have been early evidence of the mellowing-out the character would eventually entertain. It wouldn't be the first time a Trek character ignored the standard systematic formula "Star name + ordered number." It was the planet Kirk was pointing at, and Spock spoke of it in that context. Carol Marcus's response to the Reliant's Hail, "This is Regula-One, go ahead" is perfectly acceptable as well, since anybody at Regula I would almost certainly be at that lab station or its cave facilities. This much makes perfect sense to me. As for other planets in the system, we simply have no information but it seems at least highly probable. More likely than not though, the film's writers were not giving any of this too much thought beyond the basic plot needs.
Now the Mutara nebula is where the shaky really begins. It was hardly like any typical nebula astronomers have studied from Earth up to this time. Relatively small and compact, the nebula was particularly dense and energetically active. It was likely named Mutara for a reason and we would eventually learn the sector itself was in fact known as "Mutara Sector" (even though sectors were not so well defined that early in Trekdom.)
We know that Regula-I was not inside the Nebula itself, since from certain angles the Nebula would fill the entire background, but from opposite angles there were only the stars and black of space. Though it did appear to be a mere stone's throw away from Regula-I at impulse power, the minutes' travel time from the planetoid to the outer edges of the nebula might have been artistic license for the pace of the film. Nevertheless, it is very possible that the star "Regula" may have been part of a binary (or more) system with a star that had recently undergone its early (astronomically speaking) death throes, thus creating the nebula. This could explain the compact nature of the nebula. It is also highly possible that the Genesis planet was a transformed Regula-I (though I had never personally considered it) and this would better explain where a perfectly suitable star for the warm Genesis planet would have been derived; It was the same star all along. The effective range of the Genesis wave was never stipulated, but the fact that David knew there was no chance of escaping its effect at mere impulse suggests that it may indeed have been powerful enough to reach out and touch Regula-I with life.
Still, the Genesis Device was exploded deep within the nebula and when all was said and done,
nothing remained of the nebula. Surely its component matter had to have factored into the eventual formation of Genesis itself. Before the planet was revealed to be a lush garden for Spock's coffin, it was still a smolding orb coalescing from a large hot gaseous cloud. If Regula-I was indeed the seed for the Genesis Planet, the Mutara Nebula almost certainly provided fodder for its development. Any other interpretations might still be valid of course.
The Genesis Device itself may have indeed been flexible to a degree. Since we don't know how much information had been squeezed into the memory banks (indeed we have no idea how it even worked) we don't know just what its programming and subroutines might have entailed. Since the device was a detonation based technology it may have been programmed to scan the environment it was in to calculate the yield of its detonation in order to satisfy the ultimate needs of its matrix; a class-M world. There is no question the device was extremely complex in order to do what it did, failure or not, but the moment it detonated all bets were off and the avalanche would fall where it did.