Let's take a moment to consider how stories where the Doctor interacts with his past selves work. In the past, I used to think that after each multi-Doctor encounter was over and the earlier Doctors returned to their proper places in space and time, their memories of their future would be wiped to ensure the survival of their future history. This would explain why, at the end of The Five Doctors, Borusa is still imprisoned in Rassilon's tower when, if the First, Second and Third Doctors knew of his impending corruption, he would not have been allowed to get that far in the first place. But earlier on in the same story, when the Second Doctor is talking to the Brigadier, he casually mentions the time they fought Omega, which occurred in The Three Doctors. If the Second Doctor retained his memories of that event, how could it still have happened as we saw it (The Third Doctor would already have remembered everything that would occur)?