I think the only thing that hurt Serenity was the fact it was a continuation of a series that not everyone knew about. If it was just a straight action flick or pretty much ignored the series and just went with intro'ing the characters and the universe and all, it might have done a bit better... I mean i thought Serenity was an amazing film, well paced, good writing, and felt like the series could continue if they wanted to. It wasn't like the X-Files film which just felt like a 2 hour long episode with a few larger set pieces. It felt like a movie.
So I dunno I think Whedon could be a good choice to be honest, plus he's good at handling actors who are rather big ego'd, like RDJr and Chris Evans. He won't just kiss their asses, he'll make 'em work.
My only only only hope is that if they do go with Joss Whedon they limit the "whedon-verse" references. For one thing Marvel doesn't own the rights to the Buffy/Angel/Firefly/Dollhouse franchises so they couldn't use names and such, plus I'd rather my easter eggs come from the characters not from another TV show.Although I wouldn't mind a few of his stable of actors... providing they can embody that character and not just be "Hey look it's Castle!" or "Hey look its Spike!" or "Hey look its..."
Its a fine rope, but who knows... Whedon has written for Marvel before on the X-Men and did a pretty fair job, so it wouldn't shock me if they did tap him