If you look at it, the creation of Genesis doesn't make any sense, even within the rules created for TWOK. If we are to believe the proposal recorded by Carol, the Genesis device was meant to convert a dead spatial body into a living one, not convert any floating matter into a life sustaining planet. Otherwise, they could have tested the damned thing anywhere on any piece of crap ship they were going to junk.
Which is why it's difficult to believe the device would have worked on the
Reliant alone, or on the ship and the surrounding nebula. Genesis wasn't flexible - it required complex programming, and even the Marcuses weren't able to "cram another byte" into it when they prepped it for use at the Ceti Alpha desert world.
OTOH, Genesis didn't require for the target to be devoid of life - the Genesis effect was specifically stated to be a risk to our very much predeceased heroes. The experiment involving Genesis was the thing that required an utterly dead target, because that was the only way to unquestionably prove the ability of Genesis to create life (and probably the exact sort of life the designers wanted).
The ultimate aim of Genesis was to create farming worlds against UFP-wide famine. Possibly a Genesis detonation would have been able to create fields of edible plants on a dead moon, in which case the planet we got in ST3 indicated a partially failed experiment with its rather weird animal and plant forms. Possibly, though, Genesis would at best have been able to create an uncontrolled and uncontrollable jungle (which we saw both in the cave and on the eventual planetary surface), and the farms would have been created later, probably by burning down the forests. In that case, the inflexible device probably did exactly what was asked of it in the Regula system. I personally think this means it acted on a planet, one close to the detonation point. And we never heard of any Regula II or Regula III, plus the Regula we did hear of wasn't called Regula I (the station was), so perhaps the system had no other planets to go by besides the ball the station orbited - meaning it became Genesis.
Of course, even in that interpretation, we might argue that the star system did have several planets, but the scientists had chosen to give each of them a proper name. Their little rock in the sky was Regula, but also something like Mutara VI, while somewhat closer to the star the system had Dracula, or Mutara V, and somewhat farther Spatula, or Mutara VII.
Timo Saloniemi