The Ba'ku exiled them from their village, that was all. Then the Son'a left the planet because they didn't want to live in the Briar Patch. That one is explicitly stated in the film.
Colony Jarod, he said Colony. If it was just the village, why does Ru'afo make the case of being exiled to die slowly? If leaving the planet was his choice, why didn't anyone bring that point up? Even Picard says "The children have returned to expel their elders. Just as they were once expelled."