When "The Five Doctors" was first being devised, Ward and Baker were still a couple, and the hope was that they would appear together. Then after their break-up, Ward's involvement was set aside and the plan was for Baker to appear with Lis Sladen. Then Baker dropped out, and the existing cast was re-organized. In the early script Allyn mentioned above, it was the fourth Doctor and Sarah Jane who went to Gallifrey, investigated, and discovered Borusa's villainy (with a brief tease that the fourth Doctor himself might be the villain), while the second (with Jamie), third (with the Brigadier), and fifth (with Tegan) Doctors were the ones who used the three entrances of the tower, and the first Doctor stayed in the TARDIS with Susan and Turlough. Then Baker quit and Frazer Hines turned out to be too busy with Emmerdale Farm
for a full role, so things were shuffled to the version we got.