Actually, I'm pretty sure the explosion of Reliant was pretty much just a matter of the Genesis Device tearing it apart at the moment of activation. Doesn't take an explosive to do that; a turbine with a couple of missing fanblades will do the same thing to your average submarine.
Yeah, but submarines aren't carrying something as volatile as antimatter. When Reliant was torn apart, the antimatter came out of its bottles, and would've reacted with the ship and the nebula.
Genesis pretty much HAS to be a self-replicating swarm device in order to keep working under those conditions, though. The effect would have been localized to reliant and a small portion of the nebula to create a miniature genesis cave. But a swarm of devices acting on some kind of final configuration program would find the pulverized material from Reliant lacking and would set about tearing down the nebula to satisfy the rest of their mass/energy quota.
Are we assuming it was Mutara and not Regula? I always figured Mutara as a young planetary nebula (it's gotta be, to be so dense), with Regula a planet maybe Mars or asteroid belt-distant from the resulting white dwarf star. The nebula was cleared locally by the Genesis effect (and the Reliant explosion) and most of it wound up being pushed away.
It's true, this isn't fully with the appearance of Genesis in TSFS. (Interestingly, I believe it might actually be reasonably consistent with the appearance of the Baku system in Insurrection.

) But for all I love TSFS, Genesis is retarded and I'm willing to throw up my hands and declare VFX and continuity errors on that.
At any rate, I'm fine with Genesis as swarming microscopic replicator devices. It doesn't answer everything, but it's more plausible than the three step "1. explode 2. ??? 3. biosphere!" process that TWOK's visuals suggest.
Back to red matter, maybe it has something to do with chromodynamics, given the name. It's probably just as stupid, in the final analysis, but a material that does something to increase the range of the strong force to work between adjacent nuclei, and not just within nuclei (and in a really permanent fashion only within protons), would, I suppose, make extremely dense region of matter.