(Although there'd still be room for Spider-Man, I think, which is the only crossover I'd really like to see with Avengers.)
Yeah, there's no problem inherent with Spiderman and the Avengers existing (aside from Spiderman's great responsibility would probably diminish if there were lots of others available to do the same job). I tend to go with three camps: more "down to earth" heroes like Spiderman and Daredevil, more "larger than life" heroes in the Avengers, and the X-Men. Fantastic Four would probably be in the middle category too (what with fighting planet-eating aliens and all that), although I don't like two competing teams wondering around. At least keeping the Fantastic Four away from Spiderman and Daredevil will help prevent New York City from having too many superheroes, though.
I enjoyed the Spiderman crossover with the X-Men in the 90s cartoon. It worked for the episode, but it didn't work if you start to think about how common mutants are in the mutant universe. Christopher more or less explained why. It only works if he entered a bubble where the people he met only existed so Spiderman could meet them.