I don't think the Avengers subplot ruined the film. I think the story overall was blah. They didn't follow through on Tony's alcoholism which would've made a more compelling story than the blood poisoning. Even though I enjoyed seeing War Machine, I think they should've followed the comics a little bit more faithfully and had Rhodes take over as IM and then have a recovered Tony have to fight him to reclaim the title.
Also, I think they didn't know what to do with the villains. Vanko sat around for most of the movie tinkering and the decision to make Hammer an ineffectual Stark wannabe made him more of a joke than a threat. Also, the Armor Wars stuff I would've held back until the third film.
The movie wound up being just okay for me, but it could've been so much more. It was a disappointment in comparison to the first IM film. Some of the Avengers stuff actually kept my attention in IM2 more than if it hadn't been included.