I think the only irreplaceable characters at the moment for Avengers 2 is Iron Man, Cap, Hulk and Thor.
While I don't want to see it happen, Widow and Hawkeye can easily be replaced by new members.
Also Fury can be replaced with Agent Hill as director, if Jackson plays hardball.
They really should try to get everyone back for Avengers 2. Avengers 3 can get any almost entirely new team as long as they have Cap or Iron Man.