Why? The Annoyance Factor, which must be a specialty for Ethan Phillips. His character in the film The Island
had some Neelix qualities too.
As for your choices...
-B4 wasn't around enough to be terribly annoying and he wasn't even the worst thing in Nemesis anyways.
-Young Wesley... sure he was annoying, but I found it more annoying they let a kid run around on a starship... think the navy would let my 15 year old cousin play on a nuclear sub?
-Kim... he wasn't annoying, he was pathetic.
-Chakotay... he wasn't annoying, he was boring.
-Rom.... yeah he was annoying at times, but he had the best character arc out of anyone you named too.