I'd rather see the Empire done away with entirely. The Galactic Empire itself only existed for 20-odd years anyway, so why would it still be a thing more than 20 years later? As a political institution it was centered around Palpatine personally, and I do not see it surviving past him, unless Leia or Mon Mothma took over as an Empress and kept the power structure intact.
Palpatine may have been the 'glue' that held the Empire together, but it was still such a vast, galaxy-spanning organization that I doubt it simply vanished immediately after Palps' death. An organization that huge is going to take time to die.
Yeah for years I had a problem with as well, but oncve you understand that it is a fantasy it's easier to live with the sudden end of the Empire in ROTJ. And in what movie does it state how big the Empire was anyway?
Presumably the Republic straight-out became the Empire - and that's an awful lot of Senators for it to be small.