I suppose in The Five Doctors, they all remembered being One and knew that he was a fairly impressive character. Whereas, for example, even though Three remembered being Two, he didn't seem to like Two and vice-versa. Four wasn't involved, so they couldn't kowtow to him. And Five was still quite new at being Five, so perhaps didn't have the confidence to dominate the whole thing.
But yeah, generally it doesn't make a lot of sense.