Ben's "memories" don't change.
The past and reality change every week, from his muddling about, but he's still the same dude from before he first leapt, no matter what freaking shit he has done to time.
Changing the past does not change Ben.
Everytime they have changed the past, in every episode, Hannah's reality and memories of her reality have changed with the timeline.
So two horrible possibilities.
1. Ben is only in the pasts that he remembers. Everytime he changes time before all the episodes we saw, nothing we saw ever happened.
2. Everytime Ben changes the past, he still leaps into the timezones we saw, but things play out differently, because there are infinite Ben's rallying across time, each connected to their own 2023 unaware of the multiverse... Which means that Hannah is always meeting Ben's who have met different Hannahs' kinda like how Marty McFly kept making out with alt-Jennifer in the revised Biff sucks 1985, even though she was no one he had ever met before, but teenage relationships are superficial, so it hardly matters.