I won't be angry if RDJ decides to move on and take Iron Man with him. People forget that he's a great actor that can do more than play himself acting like a superhero. He can still grow as an actor and he probably wants the challenge right now. It's best to let him leave at a high point so he wants to come back for cameo's. As for The Avengers, since the roster evolves over time losing Iron Man isn't a huge hit. They need to develop Captain America and Thor and establish Hawkeye (why doesn't he have his own movie in pre-production) and Black Widow while introducing new heroes like Wonderman, Vision and Scarlet Witch.