I think Whedon is very integral to the success of Avengers.
Whedon is king of nerd culture and therefore perfect for this franchise.. he's got the tools of the trade by helming 3 "nerd" shows and several other projects so he knows the audience and he knows how to write/direct.
All successful superhero movies were made by people who understood the characters and settings and understood what made these characters what they are and then they managed to capture that on film.. Spiderman, Iron Man, Nolan's Batman and so on.
The Avengers was going to be a financial success no matter what.. it was simply big enough not to fail but it could have been just a mediocre movie if someone would have directed/written it without understanding the culture and vibe of superhero comics.. the movie would then be an amalgamation of overproduced action scenes with groan worthy one liners.. it may have been a Michael Bay movie. Avengers was still heavy on CGI and one liners but Whedon inserted heart and balance into it because he understands the medium and the fans.
Whedon may not be the best guy to write/direct some social, Oscar worthy drama but he damn well understands comic book heroes and knows how to write them.
I'm not too worried by what he said at Comic Con.. the man is currently focused on the Firefly anniversary so let him finish that and then the real talks can begin (and no doubt did he already have talks about Avengers 2 with the studio.. especially when it took off like a rocket both financially and fan approval wise).