I think it starts back as far as "Twisted".
Neelix is very condesending toward her in much of the episode and you can clearly see her starting to get pissed off by him. It gets worse later in the series when he keeps insisting shes flirting with Tom, she says she not interested in him but Neelix wont believe her. The trust between them is already starting to breakdown. However, I dont know why they waited until "Warlord" to do it. Wasn't finding out in "Fair Trade" he used to smuggle narcotics and stealing and lying to everyone enough for her to bounce?
However, going from a homeless ex-drug dealer to a traitor womanizer like Tom to the dude in "Darkling", just tells me Kes had really bad taste in men and would have probably ended up either in jail or dead from a phaser wound in an alley on some planet.