So the Doctor has 'retired'... because of Amy and Rory's departure?
Geez, did he go and read Twilight or something?
Not departure - DEATH.
Complete with tombstone. This was exactly
what the Doctor feared most about Amy/Rory, what he'd abandoned them on Earth twice to prevent. That they went on to live out a full and happy life in America doesn't even register with him. He'll never
see them again now, in any form. And his only remaining link to either, Melody/River - well, her time with him is fast drawing to a close too. And, of course, he blames himself for all of it. Five never fully recovered from Adric's death - neither will Eleven here.
Not to mention what's still hanging over his head, something he fears even more than his 'death' last season - the fields of Trenzalore, the "Fall of the Eleventh," the Question that MUST (never) be answered. (And he never did find out who blew up the TARDIS, did he?) He was already in a bad mood going into the events in New York; now after all this, and with his erasure from the very memory of the universe, it makes sense that the 1,300+ year old (at least) Doctor would decide "I've had it. I'm getting too old for this ****."