The episode was one huge push of the reset button with no consequences... again.
It's implied that a sunny day and a wife with a pretty smile at the very  least slow down Annorax long enough for Voyager to pass through. Even  that really doesn't make sense though. When Annorax has said he's been  at it for hundreds of years, it's obvious the vessel hasn't been  experiencing time as normal, and it's implicated they can move forward  and backward somehow, since he's manipulating the same people and events  over and over again trying to produce a better result.
So even if he was working on it while Voyager flew through and built it  later, he's still fly it back to the "present" time setting of Year of  Hell to start manipulating events as he wants them. And thus Voyager  would be caught up in the mess, directly or not. And if destroying the  timeship resets everything to how it happened before, why would his wife  be changed to influence him not to finish the work as the ending  implied?