They could have and arguably should have stopped with Star Trek IV. They were simply too old in the later films. No need for a new Enterprise. Saving Earth one last time could've been their last adventure.
I don't necessarily think the cast was too old to continue making films after TVH. I don't much care for the attitude that all our movie stars have to be young. But I think it would have been a good idea to allow the characters to progress and age more than they were permitted to do.
While I agree that they should have stopped making TOS films after IV, I also agree that there was nothing inherently wrong with the age of the crew. The cast being old was not what made STV fail.
Honestly, I think that at that point, we had TNG to blame for STV. What you describe (introducing new character dynamics, allowing the older characters to grow and change) would probably have happened if it was going to lead to something bigger (i.e. a new TV series with a new crew, or a mix of the new and old crews). But TNG was already on the air and the focus was on that. There wasn't any more "new" story for the TOS crew, so instead we got the status quo again because that's what TNG was doing. No, I agree that it didn't make any sense, and it certainly didn't help that STV was so damn bad (unless you ascribe to the "STV as Kirk's dream" interpretation that was brought up in another thread...)But then TFF comes along and what do we have? Kirk is mountain climbing. Chekov is back at the navigator's station. The Enterprise is back. Everything has returned to the status quo, and the status quo is now 30 years out of date and doesn't make a lick of sense. At least TUC had Valeris and Sulu off commanding his own ship, and tried to justify the rest by tossing in lines about the crew being called on for this one super-important mission. But it was still basically a film that represented the TOS status quo.
Bottom line: If TFF and TUC had been as willing to introduce new character dynamics as TMP-TVH had, I think they could have gone on just fine regardless of the characters' ages. But by shoehorning these people all back into their old, familiar roles (again, Sulu excepted), it made those films far less believable than the earlier four. IMHO, of course.
STV as Kirk's dream, yes, was covered in another recent thread, and I liked the concept and reality bending explanations offered there.
But I like what was said earlier about having the journey to the center of the galaxy be explained by Transwarp drive. IMO playing lip service to this technology would have helped tie this film in with its predecessors, whereas it feels left out in the cold as is.
Finally, to the OP, as a kid, I was let down by the ENT-A reveal. I was hoping for Excelsior and still feel like they (both the writers and the TOS crew) could have made that work. Heck, Sulu/Takei made it work just fine two films later.