First off, I want to say no offense to Ethan Phillips. He was just doing his job.
I look at that, and all I see is a waste of perfectly good air. Kind of like all that air Neelix was breathing his entire life; something the Vidiians quite rightly tried to remedy thanklessly. Let's not discount transporting directly him into space.
Everyone thinks Neelix left because of Dexa. This could not be more wrong, although she provides a useful cover.
Here are the facts:
Kes was less than one year old when Neelix took up with her. In second year of voyager, they mention halfway through she hasn't even reached 2 yet. That means she was no older than 6 months when Neelix met her.
Everyone assumes Kes dumped Neelix, sometime after warlord. Unseen. I say no, this was producers deliberating obfuscating Neelix's ongoing crimes. I will submit that Neelix was surprised at Kes' fast maturation cycle and let her go. She got too old for him.
Supporting this, in the episode Kes left, Neelix wasn't even there to say goodbye; with her emerging powers he wisely made himself scarce and just hoped to be spared her righteous wrath. Luckily she wasn't homocidal in her revenge mode yet, but she felt so disgusted, she flung the ship 10,000 light years away, just to get him away from her. The gift was to herself.
Neelix then got his tentacles onto Naomi Wildman. Anybody familiar with the show knows Naomi follows Kes' arc - grows and matures very quickly. And she has 3-4 years with Neelix, before they part ways.
In the episode Neelix left, one major plot point was that Naomi was getting too old and independent of Neelix. Loosely tied to this, and most damning of all, Kes felt the need to attack Voyager years after she left -- with an implication of untold trauma where she just felt she had to lash out, which we can assume she experienced on Voyager.
Conclusion:
Naomi Wildman, seeing Kes' angst, became close to outing Neelix and Voyager had to do this in an unsaid fashion. Neelix, like in the past, knew when it was time to skidaddle. He runs into Dexa and more importantly, her young impressionable son -- and finds his next victim to boot. Better yet, once Voyager is some months away, he's safe no matter how long the secret holds out. They're not gonna backtrack 3 months. Janeway would rather fight 5 borg cubes than take a 90 minute detour out of her way.
Secondary thoughts:
Neelix had a multitude of victims over the years and Kes and Naomi weren't the only ones that learned misery by crossing his path. One of the most obvious is Crewman Hogan, who fell victim to Neelix's legendary "Survival Skills", when he ordered the Hogan to pick up some useless bones at the mouth of a cave obviously housing a monster. We can only speculate what Hogan knew or how he crossed Neelix.
Another victim is Tuvok. Voyager was sadistic with that ongoing ballad throughout the series. While the anguish with dealing with Neelix would driven most people insane, Tuvok contracted his degenerative mental disease when combined with Neelix in the form of Tuvix. There were undoubtedly memories Tuvok had to suppress despite his cold logic and it afflicted him, much like the episode Flashback with the fake traumatic memory. We can only imagine how the real Neelix ones affected him.
This also explains why Tuvok made a special effort to reach out to Kes time and again in the only way he knew how, with the mental training. But cold Vulcan logic couldn't overcome their kind of hurt.
The last bit of supporting evidence is that every leading unmarried male had romantic dreams or hopes with Seven, however fleeting. Harry. The Doctor. Voyager's wooden plank and 1st Officer.... Chakotay. Even Tom Paris flirted with her for a scene. Except one. Guess who and why? Janeway had to pick up his slack.
Aftermath:
Unfortunately, Voyager only had 2 more episodes after we see Neelix leave with his latest victim, Brax.
Thanks to the Voyager crew's deception, I believe the Talaxian colonists would have had a misunderstanding of Neelix's [in]compentencies and handed him some type of leadership position -- whereupon the colony would have imploded within 3 months, consumed in a nuclear inferno.
Then the borg, ever hot on Voyager's trail, would have found Neelix crawling around within the lava, and assimilated him for the ship knowledge.
Over time, like with Tuvok, he would corrupted their mental processes more and more --- so thoroughly that the Queen herself goes haywire, and Neelix can assume his rightful mantle.
Darth Talax. [Seen here with other beneficiaries of his mad "Survival Skillz".]
I would have watched the 23 seasons of Voyager, what it took to get home, if one or two episodes per season featured Darth Talax of the Bjorn Again Bjorg trying to get his revenge against Voyager and getting his behind whooped every time.
That would have been worth over twenty years of Threshold episodes.