He would have had to have dumped Kes who he was infatuated with. Kes may have been the best thing that ever happened to him up to this point, there is no way he would dump her for the lure of tech and sex. Neelix seems
like he could be the sort of guy who needs to be attached to powerful women and do their bidding but his whole role with Kes was of benefactor and savior, this was his paydirt in that relationship. Every time that role was on shaky ground he was very threatened.