Whedon is great when it comes to working with ensembles. If anyone was going to be responsible for bringing together this rag-tag group of heroes, I'm glad it was him. He can take characters with vastly different personalities and give them chemistry that other directors might not get a handle on. For that reason alone, I'd love to see Whedon back for Avengers 2.
However, if the success of the first Avengers film now gives Whedon the resources to work on some of his own projects, I'd be all for that as well.