I really have to disagree with the all of the Marvel movies save Iron Man just being "set-up for the Avengers" part of your statement. The only movie that really suffered from that was Iron Man 2 bringing in Black Widow in a part that really didn't contribute much.
Outside of end of movie sequences, Thor's only connections were a shared supporting character from the Iron Man movies and a blink-and-you-miss-it Hawkeye cameo. In Captain America, there was the cube, which had been a common thread in the movies leading to The Avengers, but story-wise it was just a MacGuffin. In The Incredible Hulk, again outside of end of movie sequences, there was some name dropping, but again, nothing in your face. Most of the "build-up to Avengers" in these movies were nothing more than Easter eggs and the movies stand alone just fine.
I think it kinda goes beyond just the storylines and easter eggs though. To me it felt like those movies all suffered a bit from having to share the same basic style and tone as one another. They were all constrained by this grand Marvel plan, and weren't really allowed to breathe and be their own thing like maybe they should have.
There was no room for anything as bright and comic booky as Raimi's Spider-Man, or as dark and gritty as Nolan's Batman. Nothing really stood out, and it was all just a bit too.... samey samey for my taste.
Obviously it paid off well with the Avengers; I just that Thor and Captain America had felt like more than just a warm-up act.