"Most"? Hmm, that's debatable.
Hartnell had that black cap in the first episode and only a few times wore a panama type hat.
Pertwee briefly donned a black fedora in "Spearhead..." and then nothing.
I don't remember Colin EVER wearing a hat.
McGann, no hat, but his is the 'special case' for most conditions.
Eccleston didn't wear a hat.
Tennant, he wore a a straw hat in one scene in his final story when he visited the Ood's homeworld.
Smith, well, we remember the fez and the stetson mainly because River shot both of them, but he didn't wear them on a regular basis.
Troughton, Tom, Davison and McCoy, were the only ones to wear hats on any kind of regularity.