The episode was one huge push of the reset button with no consequences... again.
It's implied that a sunny day and a wife with a pretty smile at the very least slow down Annorax long enough for Voyager to pass through. Even that really doesn't make sense though. When Annorax has said he's been at it for hundreds of years, it's obvious the vessel hasn't been experiencing time as normal, and it's implicated they can move forward and backward somehow, since he's manipulating the same people and events over and over again trying to produce a better result.
So even if he was working on it while Voyager flew through and built it later, he's still fly it back to the "present" time setting of Year of Hell to start manipulating events as he wants them. And thus Voyager would be caught up in the mess, directly or not. And if destroying the timeship resets everything to how it happened before, why would his wife be changed to influence him not to finish the work as the ending implied?