With the exception of Kirk, no single character missing back then would have resulted in the end or decline of Star Trek. I think fans put more emphasis on the character of Spock than general movie audiences do.That's hard to say. The spin-offs had complete casts created from scratch, whereas continuing without Spock would have left a hole in the original cast of characters.
Then again, the cast proved they had a lot of chemistry without him in the bulk of TSFS, regardless of what one thinks of the plot.
I'd have to guess that if Spock had not returned it would have been difficult for the original crew to survive in terms of the box office. Spock was just too integral to that team.
I think it would have for a time being. Shoot, after Enterprise ended in '05, I thought that would be the end for Trek for good, but four years later we are having another movie. Different actors, but still Trek (I hope).