We can't say for certain whether or not Genesis was a failure, for all we know when the Genesis device exploded it created the Geneiss planet. So in essence it wasn't used as it was supposed to be. There is no evidence to prove or disprove that it terraformed Regula.
Plus we also know of one other seemingly successful use of proto-matter in DSN it was used to re-ignite a dead star and it seemed to work. So there is no evidence that use of proto-matter caused Genesis to fail.