As a matter of fact, Smith was 26 when cast, not 23. And what difference does the age of the actor playing the Doctor make, anyway. Mid-forties or mid-twenties - both are a far cry from the "true" age of the Doctor.
If we were going by true age of the Doctor, I would love to see someone like Sean Connery pull it off. Of course that's not going to happen, but I would actually watch that.