How about not having had the previous series revolve around the idea that you can cheat your way past any sort of paradox or unchangable fixed point in time with the aid of a handy robot?
You mean like they did in this very episode where Rory and Amy commited suicide? Only to then have it revealed that breaking that
fixed point via paradox damaged the temporal technobabble so badly that even considering trying it again could (and likely would) destroy the entire city if not the planet itself?
Because that's pretty much exactly what happened. Your inability (or more likely asinine refusal) to acknowledge that notwithstanding.
Nevermind that any desire to actually try again was whisked away once he realized that they lived a full and happy life, which was something the Doctor was already feeling incredibly guilty about ruining.