I'm fine with the idea of Oliver being kicked out if office. This incarnation of Oliver Queen was never much of a mayor, and the writers clearly have nothing interesting to say about politics or city government.
I think the idea of Oliver Queen as mayor of Star City was a bad mistake. Or perhaps this mistake was deliberate and a lesson for Oliver to learn - you know, like . . . you can't be a politician and a secret costumed vigilante at the same time.