They were enslaved by the wizards. It was mentioned as far back as Goblet of Fire and Order of the Phoenix.
“I think it depends on what they’re offered,” said Lupin. “And I’m not talking about gold; if they’re offered the freedoms we’ve been denying them for centuries they’re going to be tempted."