I can totally see where The Borgified Corpse is coming from. In recent years, looking back at Who, though, I've found a certain sexiness to both Pertwee and Tom Baker's Doctors. There's something about 3's lavish clothing, his energy and his chivalry that makes him attractive. I'm not sure what it is about 4. Although there's something about the waistcoat and boots that help!
I think though that there's a combination of him being dangerous to be around combined with his intellect that contributes to it. I was reading a Peter Davison interview in SFX recently where he was talking about the strictures on not touching his female companions (even when they were in distress) and how determined TPTB were to keep any hint of sexuality out of the TARDIS.
