Probably Chakotay.
Neelix was annoying, but at least he generated some kind of response from the audience. I always felt there was some potential in Neelix, but for whatever reason, the writers decided that he was destined to be their comic relief character (despite comedy not being the show's strongest point, and the Doctor being inherently funnier anyway).
Chakotay was guilty of the much greater sin of dullness. Most of Voyager's human characters suffered from this to some degree (the wonderfully potty Janeway excluded), but Chakotay had precisely one character trait - his spirituality - and even that was derived from his race.
You know, even that could have been interesting if it had been handled properly. I mean, how common are religious beliefs amongst humans in the future? I've no idea, because while Voyager could have explored Chakotay's status as a spiritual man in a mostly-secular organisation, it... didn't.
It's not surprising that the least popular TNG characters seem to be Tasha, Pulaski and Wesley, but I will say this... keep in mind that they were only around for the early, comparatively sub-par seasons. If Geordi, for example, left before the end of the first season, would he be remembered as anything other than a poorly thought out gimmick character? Much like Neelix, I feel these are characters that at least had the potential to be great.
On DS9, there was nobody on the main cast I really disliked, but on the extended cast, I had a slightly irrational dislike of Kassidy Yates. Also, pretty much any male Bajoran, most of whom seemed to have the Chakotay gene.