It's a twofold thing.
1. Fandom can be both shockingly sexist and racist--like at all the shit Elba got for his role in Thor. And if a man of color had been cast as the Doctor, there would be the same fucking conversations "But, the Doctor is WHITE!"
2. The TV business is risk averse--hell, all entertainment at certain levels is risk averse. It costs a lot of money to launch a new show, a new character. How many of them fail, starring both men and women?
The reality is Doctor Who, and other things created a long time ago, are beginning to show their age. Society has changed dramatically since the 1960s. Doctor Who has a franchise has a wonderfully built in system to keep up with the time, regeneration. The BBC would be fools to ignore it and not use it to be a part of ... well, the conversation.
For every Buffy, Ripley, and Sarah Conner, that succeeded how many failed? How many were tried compared to male characters? Hell, it's STILL hard to launch a female lead action franchise... How many people (men) tried to get Captain Marvel to fail? How many years did it take to finally get a Black Widow movie made?
I'm not saying that gender flipping is the only way to go in all circumstances, of course not.
But, with Doctor Who, went you literally have it built into the show, why not take advantage of it.
The BBC isn't looking at how Doctor Who was done IN THE PAST, they are looking to the future. They don't want Doctor Who to be frozen in time, they want a franchise that will continue to go on.