I've somewhat addressed this idea elsewhere, but here's my thought.
First, to make a Frontier/Colony Trek
work you need a group of regulars who remain fascinating to watch in and of themselves. In this case, there's a natural tension between the colonists per se and the Starfleet personnel assigned there. Towards that end I would suggest the colonists themselves are a people apart. Perhaps they are the survivors of a notorious POW camp in a war a generation past. Or they endured some kind of natural disaster together, the nature of which genuinely shook their faith in the Federation. Not that either group should come across as homogenous.
Second, let them be pretty much on their own. No whistling up a Quasar
-class heavy cruiser every time trouble rears its head. Let them be far away, isolated enough they must work together.
Third, the colony itself should present a mystery, a story in its own right. Not a single mystery either but a multi-layered one that can baffle and tease both characters and audience.
What comes to my mind is a Starfleet Outpost crewed by folks who in another era might belong to the French Foreign Legion--each has a reason they're in the middle of nowhere. Needn't be a dark secret, or even a secret at all, at least not for everyone. Likewise the colonists as builders of a new society struggle to find a compromise between various ideas--about gender roles, about justice, about history. Yet not a brand new community. One that has existed long enough to have some nature of its own, yet for the younger people to feel stifled. A place and community with their own assumptions.