It's hard to guess these things, but it seems to me that a part of The Avengers' success came from having that many superheroes on screen in one movie and they all did stuff and the stuff was big
. When NYC was threatened, stuff was smashing everywhere.
As for the contributions of Joss Whedon Quimself, his script I though mediocre. I didn't buy into the Black Widow myself. And everything Fury did was downright stupid. Thor wasn't handled properly. Tom Hiddleston managed to make Loki interesting in Thor, but with Whedon's script, he was just a greasy haired asshole who made stupid speeches and got bitchslapped for his pains. The discovery that Banner was lying in every scene before the climactic battle I suppose is surprising. Writing the new Banner who is always angry and is in total control of the Hulk is going to be a challenge and it's not at all obvious Whedon is up to it.
As to the direction, the big climactic action scene flowed from one superhero to the other, which goes directly to the big selling point of Avengers being the sheer spectacle of all those superheroes. For the rest? No, it wasn't very visceral. You didn't have any real idea of why anything was happening. The so-called villains in particular were just aimlessly fighting. For my part, it was the scale that was interesting.
I doubt I'll purchase The Avengers, despite the nostalgia value. And if they hire someone else for directing the sequel it won't lower my expectations a bit. They're already lowered because sequels are rarely as good as the originals.