I dont know if it was the 1 Dalek that scared him, it was what it represented, the survival of the Daleks, and a reminder of his actions in the Time War, a reminder of the end of his people.that is worth quoting, and were probaly best shown off in "Dalek"
I agree with this. The best handling of the Daleks was in the title episode. Why? Because one Dalek single handedly caused widespread terror throughout a concealed facility with the threat of unleashing its power on the outer community. The Doctor was in panic mode from start to finish, over 1 Dalek. Come millions - meh.
Shell shock, PTSD, that kind of thing