It's a complicated question and a complicated question. Your end statement is not necessarily true either...look at Brexit. About half the country jumped up and down about 'sovrinty init' and half didn't give a monkeys as long as things were generally stable and OK. The history of the British Empire is full of that sort of distinction too, not least as it tended to favour one side in a preexisting setup before nominally 'taking over' (not denying we sent some nasty buggers out into the world, and the Empire was decidedly not a bed of roses for anyone at the bottom, including in the British Isles themselves.) or changing the setup in their own favour. Rule in India came from taking over an extant setup....rule in Hong Kong started with basically an empty island that local people then came to...some of the Caribbean islands started off by being 'liberated' from groups that had not ceased slave trading, Canada was a comparatively 'free' place to live compared to North America at that time. The whole history is full of contradictions, and colonial history is full of those shades where it very much depended on whose warship turned up, whose side they took, and right down to whether the crew of that ship were comaparatively progressive or not. Nasty business overall, but humanity in general takes a long time to stop hitting each other on the head or claiming rule over other humans. (We still haven't) After that it's all just history and how local people see their own history and future. The British Empire basically modernised itself out of existence, becoming the commonwealth, which is a very different thing to an empire, though we are all taking a long time to shed the shadow of Empire. Once upon a time my ancestors were busy dying of various diseases and being poor as it gets in the east end of London a few generations back, invisible to those above...once upon a time my friends ancestors were in a very similar situation on an island in the Caribbean...the 'rulers' didn't necessarily have the right to do whatever they wanted to our respective groups anymore, but it took a long time for that message to sink in. Then his family came here for what they saw as a chance at something better, not least due to the societal reforms that were happening as the result of things like the tragedies of the two world wars, and slowly but surely we ended up better off than our forebears just a few generations back. Sure, we don't have the same skin tone, but we grew up on these same streets together, and thanks to improvements made in our own parents lifetime have almost the same opportunities...except when that ugly shadow tries to come back out in some individual somewhere. Like those muppets who worry about the Melanin content of Prince Harry's girlfriend.
If you take a longer look at history, you will also notice that Britain is indeed still living under the rule of the conquering families that made it part of an Empire about a thousand years ago, and the results of wars more recent. Humanities best bet is to just drag itself forward over and over again, until we escape the shadow. (Even the EU is essentially a club of ex-Empire nations, taking turns ruling each other instead of conquering the world, and unsurprisingly when one group gets to much power for too long, then the trouble starts. It's still a better solution than warships and violence mind you.) That goes for white nations, black nations, yellow nations, red nations, blue nations, tartan nations, nations with stripes, nations with polka dots....sooner or later we have to stop hitting each other on the head, standing on the faces of those below us to scramble up some kind of ladder, fighting those beside us in case they get something we don't have.
History can be a gift, or a curse, it's up to us to do something positive with it...and I think that's what some of these nations have chosen to do. To use the positive, acknowledge and attempt to repair the damage of the negative, and move forwards recognising a sort of familial bond through a shared past.
