Correct me if I'm wrong but Empire comes out of the Special Editions pretty much unscathed, doesn't it? I struggle to think of an addition or change that gets people as angry as the "Greedo Shooting First" thing or that's as annoying as the "Jedi Rocks" musical interlude.
The worst "update" to Empire was the recutting of the scene where Vader returns to his ship while Luke is being rescued by Leiah in the Falcon. Lucas' assertion that Vader simply saying "Bring my shuttle" wasn't enough to explain how he got to his destroyer was asinine. And the additional shots of Vader's shuttle taking off and going to the ship, intercut with Luke's rescue, utterly destroyed the pacing of the previously well-cut and tense scene.
I wasn't really happy with the added windows in Cloud City either. All the ship-zipping business out there distracted from what we were supposed to be paying attention to, and broke up all that beautiful white-and-gray set design.