They had a food fight in Paturation.
Kes and Neelix were still together at the time.
Apparently Neelix's friendship was more important to Tom than stealing Kes off the kitchenrat.
Most people assume that the Break up in Warlord that happened on screen while Kes was possessed, stuck. After the fact Neelix decided that the sociopath freejacking his girlfriend was probably right.
If he stayed with Kes, than Sam Wildeman and Neelix would never have gotten so intimate.