^Beat me to it. But there is no direct evidence to say that using proto-matter was wrong.
As far as we know the Genesis Cave worked.
As for the Genesis Planet itself, was it Regula that was terraformed or did the Genesis device create a new planet. In the case of the former it was never meat to be detonated that far away from it's target, and in the case of the later it was never designed to form a planet.
The Genesis Device was in fact designed to form a planet, that was the reason it was designed to do anything at all. However, it was designed to form a planet out of a planet, not out of a nebula, so even without the protomatter, it would have been unstable.