Prisoner Zero was just talking about the fact that the cracks came from the Doctor's own TARDIS blowing up when he said that, finding it amusing that the Doctor didn't know it was his own fault. Prisoner Zero couldn't make people forget him as soon as they looked away like the Silents could, so that wasn't what Amy saw in the TARDIS.I somewhat suspect it was Prisoner Zero. The voice is similar. Also that might have been his crime. Blowing up a TARDIS. Time and the Doctor don't go in order when it comes to the Silence and Trenzalor.
I never really considered much about Prisoner Zero beyond the one episode, but I kinda like this idea. It would certainly explain why Prisoner Zero seemed so proud.
"The cracks in the skin of the universe. Don't you know where they came from?" *cracks an evil little smile* "You don't, do you?"
This really isn't the big mystery some people have made it out to be.