Well, logically this would mean that Harry was a two-timing asshole. Of course, that's not what was implied, since both the writers and Wang forgot about Libby when they wrote the episode about Lyndsey.
OK, here we go:
In this case, Harry was actually a "two-timing a**hole" who had a secret relationship with his Academy colleague Lyndsey Ballard while dating Libby at the same time.
When Libby did find out about it, she was furious and threatened to phaser Harry to "another existance below the surface of Earth". Harry became very scared and used his contacts in Starfleet to be stationed on Voyager where Lyndsey was instead of Danny Byrd who actually got the job before becoming food-poisoned and unable to report for duty on Voyager. It's always good to have contacts at the Academy. Maybe it was Lieutenant Lasca who helped Harry with that detail.
However, on board Voyager, Harry discovered that Lyndsey had found out about his dabbling with Libby and decided to break up with him. And not only that, she also told Captain Janeway that she didn't want anything to do with Harry under any circumstances.
In order to avoid problems, Janeway ordered both Lyndsey and Harry to stay as far away from each other as possible. When Torres became Chief Engineer, she was informed about the situation and with the help of Carey and the other Engineering crew, they all did their best to keep them apart. So when Harry entered Engineering, Lyndsey felt an urgent need to go to the toilet, when Harry entered the mess hall, Lyndsey was hit by nausea and had to leave and when Harry entered Sickbay, Lyndsey left without letting The Doc examine her. And vice versa!
Since we mostly followed Harry who was one of the main characters, we never got to see Lyndsey in the season 1-3 episodes and Harry's closest friends, like Tom and B'Elanna were polite enough to keep quiet about it
No doubt that Harry was struck by doubt and guilt, therefore his experiences in "Non Sequitur" when Harry ended up in an alternate timeline with Libby and discussing the fate of Danny Byrd.
All that changed when Lyndsey was killed in 2374 and later on when "the other Lyndsey" returned to the ship.
As we all know, Harry had then been replaced by a duplicate of himself which gave Lyndsey (or more correctly Lyndsey 2) a reason to forgive him.
The same when Duplicate Harry returned to Earth. Both Libby and Danny Byrd forgave him since they realized that the Duplicate Harry wasn't actually the exact same Harry who had cheated and fooled them.