Agreed. And people seem to forget that this new trilogy still comes from Lucas's original story that he dreamt up decades ago.
If Abrams was creating a new story completely from scratch (as with the Trek movies), I'd probably be worried as well, but that's not what is happening here. At the most, he and Kasdan have probably just changed some of the details of Lucas's story here and there, and improved the action and characters.
Also, critics and haters tend to forget one important element:
Lawrence Kasdan.
The man who helped craft both Empire and Jedi and make both movies as good or even epic as they are. Kasdan knows what he's doing and he's a 35-year veteran of the Lucas universe. The participation of a man who played a critical role in making two of the three films in the Original Trilogy will be nothing but a good thing for the story of Episode VII.
This. So much. The man probably had more influence on the OT than anyone besides Lucas himself. And, incedentally, he also helped write Raiders Of The Lost Ark. Kasdan's involvement is one of the key reasons why I'm so optimistic about the future of the franchise.