Phlox is one of my all-time favorite Trek characters, and certainly my favorite on Enterprise.
I blame a lot of Neelix's irksome nature on Philip's acting, which I consider mediocre at best. He was Particularly annoying through most of the first season until at least "Jetrel". His portrayal of romance with Kes wasn't believeable in the least (I KNOW I can act better than that) and it really got bad when he was overcome with jealousy towards Paris. Add to this the fact that the character was particularly useless by the third season, and you have a character thats near the bottom of my favorites list.