I see no reason why the events during the journey should neither be remembered nor have an affect on later stories. Life may not be as tightly structured as a carefully constructed serialized story, but it's certainly not a collection of incidents that are immediately forgotten and without after-effects. Unfortunately, that describes the vast majority of the stories told on the series. There's little progression from beginning to end, as evidenced by the way the production order was sometimes reshuffled when the episodes were aired (something you wouldn't know if production information hadn't been made available).
An individual's life is not structured at all. It is usually an act of imagination to superimpose a narrative on it. It is in fact such a truism that most people will stenuously deny that there are any patterns or narratives in social life, i.e, history!
Life usually does seem like of collection of incidents that are immmediately forgotten and without after-effects. Usually it is only the passage of time that leaves the marks of change. Genuinely life-changing events are those which cause a change in how daily life goes about, such as marriage, parenthood, to a lesser degree things like major illnesses or new jobs. People who like to spend a great deal of time discussing their personal history are well known to us in daily life however. They are usually avoided as being extremely boring.
So far as Voyager is concerned, note first how irrelevant most of these life changing events are to the characters in a show with Voyager's premise. In any event, despite the lack of emo about personal life, characters like Torres, Paris, the Doctor, Neelix, even a minor character like Kim just didn't act the same way at the end as at the beginning. It was in fact the characters who underwent the most drastic changes whose changes were the least plausible, namely, Kes and Seven of Nine.
I wonder if what's going on with such an objection, that has so little rational basis, isn't trying to express something deeper, which is that some viewers want the show to be about the emotional lives of the characters, to present the so rarely found in real life narratives about emotional growth, neatly dramatized and self referential, which is to say, self absorbed.
The unspoken hope, even to oneself, is for a character who can vicariously live out the daydreams. The constant recurrence of tacit or explicit references and comparisons to DS9 happens because DS9 ended up committed to that kind of character "development." The number of characters on DS9 who ended up being exemplars in one way or another of Freud's "family romance," is a good example.
By and large, Voyager was about what happened to ordinary people, or even about somebody else, not what about what extraordinary people did in a grand adventure. Yes, technically, just getting home would be a grand adventure, but the people never really seemed to be satisfyingly special.
No, the show never really gets any better. Even Seven of Nine gets cut down to size in the end.