First... It takes 6 pilots to steer a TARDIS properly.
That used to be fanfiction. I just about came when they canonized that tidbit.
The first Doctor's problem was that he couldn't be on both sides of the console like younger versions of himself managed to... Ben and Polly were trying to get home, and they got dropped off in a relative fraction of the time it took the first Doctor to find when Ian and Barbara belonged, but the others, Victoria, Jamie and Zoe were all in for the duration.