Then again... in Time of the Doctor, the Doctor spent practically the exact amount of his 11 lives in Trenzalore, so he has been away from the TARDIS as well, and for a long, long time, as well.
In retrospect, I find it difficult to accept that "Time of the Doctor" happened. I mean, obviously it did. Clara watched her friend age before her eyes, and the Doctor regenerated. But the twelfth Doctor never acts like someone who spent 900 years trapped on an alien planet, and he treated Clara from the start as though she were someone he knew recently rather than someone he knew 900 years earlier (excepting two days over the span of those centuries). It's the inability (or unwillingness) of Moffat and the eighth season to treat the aftermath of Trenzalore realistically that forces me to question the whole Trenzalore experience.
You just pinpointed why I don't like series 8 all that much. So much potential to play with a Doctor who's practically exploring the universe anew, again - like his first incarnation, as an old man out to see the universe after centuries of confinement. But fuck that - out for some coffee!
That said, he did have some memory gaps, and subconsciously choosing to be an old man in his regeneration seems to have been a reaction to how old he feels as the Doctor at this point. But Moffat didn't really go anywhere further with it.
Maybe Big Finish can realize that gap with the potential you mentioned, some day....
EDIT: Christopher and Wormhole offered some good explanations for this, though... so I guess it was not a complete waste.