There is nothing remotely ageist about saying that most actors above a certain age can't handle the rigors of the new series' production schedule. (Of course, there are some who could, but the likelihood that one of them would be what the producers are looking for in a new Doctor is even slimmer.)
Certainly it is ageism to assume because someone is of a certain age range that they cannot handle the "rigors" of the production schedule. If there is no other reason to not include them other than they're "old" so they wouldn't be able to handle it, then that's an ageist view.
Of some note, I copied this from Wikipedia:
In a survey for the University of Kent, England, 29% of respondents stated that they had suffered from age discrimination. This is a higher proportion than for gender or race discrimination. Dominic Abrams, Social Psychology professor at the University, concluded that ageism is the most pervasive form of prejudice experienced in the UK population.
And that's the only reason anyone has given for casting 40ish actors- nothing about who could own the role or playing elderly or playing young or whatever.
My response concerning assumptions that old equals grandfather-figure was in response to this post:
Because the show is appealing to a younger demographic...who really isn't interested in a Grandfather figure anymore...