So until the Watcher uses his Radio Shack device to erase Peter from time*, both Peters did exist, and Walter did cross over to steal Alt-Peter (the one we've seen for the last three seasons), but fails to save him. So you still have the foundation intact for why the Walters distrust each other and all that, you just don't have Peter involved in anything past the kidnapping. At least that's my interpretation.
That's not a bad theory; it does explain some of the issues I had with the current set-up. Quite likely you are correct, and we'll learn in the next few eps that both Peter Prime and 2.0 did exist, but died as children.
That's brilliant, really. I'll go with that notion until we see something that seems to suggest otherwise.