Right. People like to invoke the "infinite multiverse" line as an excuse for any random nonsense, as if that somehow made it scientific, but in actual scientific practice, if your equation turns out an infinite result, that's usually evidence that you've done the math wrong. In practice, if we're talking about divergent timelines, we're not even really talking about the whole observable universe -- just about the ensemble of particles that have been able to interact with each other in the time since the diverging incident occurred. So in a physically realistic universe, that would mean a sphere n light-years in radius for a divergence that happened n years before -- although the sphere would be rather larger in a universe with FTL phenomena like subspace and tachyons and stuff. Either way, though, it's an even smaller finite number of particles, and the set of possible configurations would be commensurately smaller as well.