Star Trek's premise is that we pretty much overcome nationalism and racial distinctions ...
Actually we do see nationalism. Scott, Chekov, O'Brien and Reed all display nationalism at one point or another. During the first season of TNG, Picard speaks openly of his nationalist feeling about being French.
There seems to be an implication of planetary unification ...
There are on screen references to a HMS Lord Nelson in the year 2120 being on a mission of "deep space" exploration and the HMS New Zealand in 2135 undertaking a diplomatic mission.
HMS (Her Majesty's Ship) sound more like a starship registered out of Britain, and not "Earth."
There's also a VK Yuri Gagarin on a colonization mission in 2105, "VK" iirc is Russian.
For instance, when humans began colonizing planets, would an ethnic group like Han Chinese claim a given world and violently repel any other nationals?
I can easily imagine nations (and also private groups) engaging in large scale interstellar colonization following the discovery of warp drive. That said, I've alway had a problem with a group claiming an entire planet base solely on their establishing a colony on the surface. A reasonable area or even a region sure, but not an entire planet. Not that that wouldn't prevent a group from employing violence (or it's threat) to defend their claim.
Would seperatist groups like the Haredim attempt to create their own private society away from the crowds and dangers of Earth?
Sure, groups would leave Earth for a variety of reasons. And not just to travel to "virgin" worlds either, they could go to established (by other species) colonies world, and even to alien homeworlds.
To do the jobs that the aliens don't want to do.
Would governments use newly-accessible planets to export their prisoners?
Penal colonies are a Star Trek tradition.
.