Yeah... Harry Kim bothers me waaaay more.
And he's in every episode!
I would say Neelix is the most annoying thing. Harry's the the poor schmuck that everything bad happens to. Kinda like Voyager's O'brien.
Except where you felt sorry for O'brien's suffering, you just kinda laugh at Kim's.