I think they went into the past of the Prime universe and altered it. Just as the Enterprise C altered events by going back in time.
I think you are misunderstanding the temporal mechanics involved.
Yes, Spock going into the past and being captured by the Romulans is just like Lt. Yar going into the past on the Enterprise-C and being captured by the Romulans.
In both cases, their travel to the past resulted in a NEW, ALTERNATE TIMELINE that was DIFFERENT from the one they had come from.
In Yar's original timeline, the Federation was at war with the Klingons. By going back in time, she created a NEW, ALTERNATE TIMELINE where the Federation was not at war, and in fact a Klingon officer served on the Enterprise-D.
In Spock's "original" timeline (in reality, his last 40 years were in the alternate timeline created by Lt. Yar on the Enterprise-C) the planet Vulcan still existed, Kirk's father had seen Kirk grow up, and Spock was estranged from his own father. With Spock and Nero going back in time to a NEW, ALTERNATE TIMELINE, they changed all the events that they remembered from their own timeline.
When you talk about "altering your own timeline," you're talking about a "Back to the Future"-style paradox where you start fading away from photographs of yourself as you erase your own past.
Lt. Yar and Spock definitely didn't fade away. They were real people from real timelines with real pasts, who both happened to be in a NEW, ALTERNATE TIMELINE creating new events that conflicted with those they remembered.
All timelines are equally real. Just because they never showed what happened after Yar or Spock disappeared into those "spatial anomalies," that doesn't mean that those timelines ended at the moment those people entered a NEW, ALTERNATE TIMELINE. It's just that Spock and Yar were the point-of-view characters for those particular stories, so what happened in their original timelines after they left wasn't relevant to their personal stories.
Imagine in both cases if Picard was the viewpoint character. In Yar's original timeline, Picard would have watched the Enterprise-C disappear into the rift, and then would never hear from Yar again. He would continue fighting the Federation-Klingon war without her. From his point-of-view, only Yar and the Enterprise-C disappeared, but everything else was unchanged. But from Yar's point-of-view, her entire past was gone forever.
Or imagine that Picard on the Enterprise-E would see Ambassador Spock disappear into the black hole and would never hear from him again. But he would help re-settle Romulan survivors to new colonies after the supernova and get on with his life.
The point is that what you, the viewer, believe to be a "single timeline" is actually the personal timeline of the viewpoint character in a particular episode, who moves from one timeline to another, but remembers both.
You saw Yar and Spock changing history, then remaining in that new timeline, so for those specific people, yes, it seems like they have "altered" their "own" pasts (i.e., the events they are witnessing are different from the ones they remember), but for everyone else in the universe, events have only happened once, with only one outcome, whether they are in the original or new timeline.