The Genesis Device is a piece of handwaving -- that's commonplace in SF -- but it is set up with reasonably clear / intuitive in-setting rules and more-or-less plays by them, particularly in its initial film appearance*, which Red Matter is not and does not.
Are we talking about the same Genesis Device that was designed for existing planetary deployment but ends up absorbing all the matter in a nebula using space magic to construct a habitable planet (or to buff up Regula a bit, depending on who you ask), which fortunately just happens to be in the Goldilocks Zone of a star compatible with humanoid life? Or that can create a tropical jungle in a cave, complete with weather, but doesn't extend beyond the precise boundaries of said cave (not even through passages with no doors) despite being depicted as an expanding energy wave that absorbs and reforms matter? And did it create that little artificial Sun in the Genesis Cave, or was that Starfleet issue? And if it was Starfleet, why didn't the Genesis Wave eat that?
In what way was that different from Red Matter's silliness and alleged inconsistencies?
I for the life of me couldn't work out why red matter needed to be drilled into some things but not others...
How would one go about drilling into the heart of an expanding supernova (or the Hobus Star itself before detonation)? Ideally deploying the Red Matter to the center might be the fastest and most efficient method of absorbing the entire thing, but that wasn't a survivable option, so old Spock had to deploy it into the blast wave. Still gets the job done, just not as quickly or evenly.
Why would one need to drill into the center of a giant Edward Scissorhands looking ship when with that design you can pretty much fly right into the middle of it like kamikaze Spock did with the Jellyfish?
The two times it was used against planets, they drilled down to the planet core. I see no inconsistency. Three different types of targets, three different methods of deployment dictated by necessity, but consistent when used against the same type of target.
As far as why they drilled to the core of the planets, I just figured that if you drop it at the surface, it would possibly allow people on the other side of the planet more time to takeoff or be rescued, whereas if you drop it into the core, the entire planet start collapsing in on itself evenly and immediately (not to mention other devastating effects removing the core would cause, like eliminating the planet's magnetic field and irradiating the surface), making escape more difficult due to the increased speed of destruction. Since Nero was bent on genocide, not just destroying the planet, he'd want to limit the amount of survivors as much as possible. Dropped on the surface it would still get the job done, just not as fast.
why the amount of red matter seemed irrelevant to the size of the black hole produced...
I just figured that the Red Matter simply expanded gradually to the size of the object it was absorbing, regardless of the amount used.
or what exactly we were supposed to assume about the black holes it produced, when they facilitate time travel and when they don't and so on. So that's not exactly what I look for from internal consistency.
I just figured that every time a wormhole is created, it's sending stuff back in time to another universe. It's just that Vulcan is being sent back in time to another universe in small pieces because the gradual expansion of the wormhole tore it apart from the inside.