I see no reason why the events during the journey should neither be remembered nor have an affect on later stories. Life may not be as tightly structured as a carefully constructed serialized story, but it's certainly not a collection of incidents that are immediately forgotten and without after-effects. Unfortunately, that describes the vast majority of the stories told on the series. There's little progression from beginning to end, as evidenced by the way the production order was sometimes reshuffled when the episodes were aired (something you wouldn't know if production information hadn't been made available).
What was the lasting effect of Deanna Troi after her mind rape beyond that ep.?
How about LeForge after being brainwashed by the Romulans?
O'Brian beyond "Hard Times"?
Sisko after poisoning a planet?
Hell, the crew of the Enterprise in "Genesis"
There have been several times in ALL the series were severe traumatic events are forgotten by the next episode with no lasting effects in any series. Voyager is no different. We accepted it then, so we gave them the green light to continue.