I think it's very possible that the trek film series would have gone on without Spock. They set up their new crew in TWOK, Saavik would have assumed Spock's role in the trio, but creating a different dynamic to it, Logic wuld no longer just be subordinate to Kirk, but to McCoy as well. I think if Spock were not going to return in III, Paramount would have given Kirste Alley what she wanted as far as her contract went, since she would be again a very central part of the story. It would have become a little more of an ENT vibe with Saavik becoming a sort of T'Pol, but that dynamic worked well enough on ENT.
I think if Nimoy had signed on to direct and not actually bring Spock back the fans would have responded to the movie and supported it. But I think there would be a required "Flashback" scene at some point with Spock in it in order to really hook them in. Sort of like what they did on The West Wing with Mrs. Landingham. After she died, they had 2 flashback episodes, both featured a brief cameo with her.
I think the wide Success of Trek IV was probably one of the most important factors in TNG happening though, so it's a real "Who knows?" situation there. They might have gone on to do IV in the same manner anyway with Nimoy just on as a director,so creativly it could have gne very similarly, with the exception of Spock's "Colorful metaphors"
Regardless, Trek would have been different, and I like th Trek Film Series the way it is. Well, TFF should be fixed. But that's not the discussion we're having right now.