They should have never been awarded, and neither should have been Beijing 2008 or the FIFA World Cup 2016 or 2020 to their respective countries.
You are forgetting that these weren't awarded to great, upstanding nations; they were awarded to the highest bidder and/or whoever puts on the best show.
Sport very rarely, if ever, promotes the kind of change people are looking for here. Hell, it took two hundred years from the African slaves coming over on ships to the new world to be allowed to use the same washroom as a white person and there are still people who think they are lesser human beings. Changes like gay marriage, hell even gay acceptance take time and will not take place because a country gets a sporting event taken away from them.
Yes the IOC and FIFA are horrible human beings for awarding these events to these places, but keeping them out will only exacerbate the situations in the short term, instead of fixing them for the long term.