It would make sense for the Doctor (and the audience) to be somewhat confused, considering that at various times he might be thinking in Earth years, Gallifreyan years, whatever length the years are on Trenzalore (as converted to other kinds of years), or whether he's thinking of how many years old he is biologically or chronologically. After all, if he stops off at one place and returns to it 500 years later (a few minutes later for him personally) it doesn't mean he's actually 500 years older.All good arguments.
though I don't think even as a milennia old timetraveler would he lose track of his rough age.
All he needs is a reliable personal clock he always carries with him. That could either be the TARDIS, his screwdriver or both.
The Doctor was rather vain in some of his incarnations.It's also possible that the Doctor doesn't want to know his exact (or rough) age. He might actively choose not to keep track so that he can lie to himself that he isn't so old.

Add Tom Baker to the list of Doctors who visibly aged during their time as the Doctor. It's not that glaringly obvious (as it was with Sylvester McCoy and Paul McGann), but if you compare Tom Baker in Logopolis to his 7-years-younger self in Robot, you can see that he has visibly aged. If you want an in-universe explanation for that, there was a time inbetween Leela and Romana when he had no Companions, so there could have been many adventures besides the one story that took place on Gallifrey (The Deadly Assassin).
The only real clues we have are the approximate lengths of time when his Companions are with him, and when the episode specifically states a timeframe. Personally, I've decided to stop worrying about it and just accept that when nuWho came along, somebody did a bit of an adjustment.