I think the success of The Avengers as a film, not as a billion dollar phenomenon that has just as much to do with expectations and franchise loyalty, has everything to do with Whedon.
His writing, and his deft handling of a film with many characters, all who were given important and meaningful things to do, was evident, and done better than other films with fewer characters...like Iron Man 2. The amazing rebirth of the Hulk, for instance, has so much to do with Whedon understanding
the character. He understood them all, and that's why it worked. His pacing, writing, shooting and editing of the interactions between the characters are what made the film amazing.
That's what's most important to me. If the action is mid-level (which I don't think it is), it doesn't matter to me. It was the character scenes that made me love the film.