From big bang (multiple big bangs) to entropy, every universe that we saw in Year of Hell took place beginning to end. Annorax was destroying future history as much as the past. Just because Captain Chakotay made it past Krenim space like a champion, it doesn't mean that Anorax was not yet about to destroy that future and replace it with another one.
After a universe normalized from a wrecking by the weapon, Annorax scans the universe past present and future to understand the state of the Empire and what he doesn't see is himself destroying time again in the relatively immediate future. the future must flow unto the end until it doesn't because as an agent outside of time he is not bound to any such continuity or predestination or paradoxes.
Although I do think that the before and After timeline is the base line original universe however, the krenim were using temporal torpedoes which means that they were on their way to inventing all sorted of time technology that could be as dangerous to every one as Annorax's weapon.
Although... Anroax would have noticed Kes and jumping about and had to deal with her since according to his book keeping Voyager was temporally inert and beneath their consideration.