Thanks for reading!
I wouldn't necessarily say genocide is out of character for the Dominion--what they did to the Teplan people would have ended up becoming genocide if the Federation hadn't intervened with a cure. But, I do think that they would rather put a subject race under their feet than destroy them entirely.
While humans CAN be pack animals to an extent, when you look at the way aliens are portrayed in the Trekiverse, humans do seem "volatile" when compared to some of the other races, in terms of being unpredictable...in a lot of ways, Bajorans are the closest comparison. The unpredictability is what freaked out the Dominion, as is the human hatred of conquest (whereas, say, the pacifist Mizarians have never resisted when their world is conquered). Humanity likely would've become either suicidally rebellious, or done something like what they actually did, just sooner. So, the Dominion THOUGHT they could pacify humanity this way.
What's been introduced to the human race, through the Graft, goes to a higher level than humanity's normal social instincts as we know them. It is true that modern humans are vulnerable to groupthink, bystander effect, and other such things.
But in comparison, an unaltered human might seem to have something like Asperger's Syndrome, to someone who DID have the Graft--to be missing another level of tone, body language, and hierarchical instinct that goes beyond standard humanity. (And that's another reason I think you might get resistance from the young, when it comes to "curing" themselves: it might feel like blinding themselves. To the older people, while their reconfigured brains can handle the input from a physical standpoint, most of them do not want it. Some might choose not to accept a cure, though, if their children won't.)