Plus you could argue that Tom, Colin and Sylvester weren't that well known before they took on the role.
Actually Colin and Sylv were both quite well known at the time, in the UK at least.
Colin Baker had one of television's more high profile "bad guy" roles during the late 1970s as Paul Merroney in the Dynasty-esque quasi-soap The Brothers. He certainly wasn't by any means a no-name when he got the role of Doctor Who.
Sylv was probably lesser known than Colin, but many people were aware of him from his time with the Ken Campbell roadshow. So much so that "Ferret down the pants" jokes were immediately made at his expense once he was announced in the role (one of his best known routines involved, allegedly, putting these rodents down his trousers live on stage).
Tom Baker, on the other hand... I'll give you that one.
For perhaps the most famous classic Doctor of all, he really *was* practically unknown by the public at large when he was cast, although he had a good reputation with the casting agents within the industry itself (which is how his name got put forward to then Doctor Who producer Barry Letts in the first place).