I liked Phlox, he was irritating in an interesting way.
As for Neelix... why did Seven have to bring him back from the dead?
Neelix is more annoying than Phlox for sure, while Phlox was a much better character. I think Neelix would've been more interesting if he had been the doctor on VOY -- I believe Robert Picardo auditioned for both roles, the EMH and Neelix.
Phlox was only a tiny bit annoying in the beginning, with his odd grins and facial tics, but I found him more useful as ENT's CMO. Neelix was just Cookie in space, and that always annoyed me.
You see how he eventually outlives his usefulness as a guide, the original intent of the character, in the ep where he meets his fellow Talaxian (Wixi-something, too lazy to look it up!) and realizes they're going into areas of space he's never encountered. It would have been more interesting if Neelix was some kind of wild-eyed adventurer, like Steve Fosset, intent on settng a record by traversing the Delta Quadrant, and then meeting Voyager
on his way back to his homeworld after decades in space.