Even early on, with "The Naked Time" you had time travel. You had time go backwards, and yet how do the characters treat it? Then they go back in time for historic research in "Assignment: Earth." Multiple people die randomly in other time travel events. Guess their lives didn't impact the timeline.Because it's all the same universe, simply altered.
The prime universe has had so many examples of time travel throughout the years, it's the only option. Otherwise, what really is the prime timeline? Every single example of time travel would equal a new universe being formed and means that we haven't been following the prime timeline since a few episodes into season 1 of TOS.
It becomes ridiculous to pull out the seams to this extent. Treating it as one, if modified at times, timeline, is pretty much my preferred way. Aside from TOS as a dramatic recreation, and completely separate from the rest of Trek.