Behaviorally the Ferengi always reminded me of ferrets or cats. Even their diet suggests this. We know the Ferengi do eat small prey--grubs, worms, snails, and so on, but I don't usually hear about their eating larger animals.
My theory is that, similar to cats or ferrets, the Ferengi evolved from predators, but NOT apex predators the way humans and presumably Vulcans, Klingons, and Cardassians did (judging from behavior). Their fight-or-flight behavior, their threat displays and distress cries, and so on, and terror of confronting larger opponents while being willing to prey on smaller ones, AND even their contention that they have never engaged in the type of large-scale warfare seen by "apex" species like the ones I mentioned before, all fit with the idea of a non-apex-predator species. And the judgments passed on them by the "apex" species definitely seem like they would stem from such a fundamental disconnect in mindsets.
I suspect that something really huge and nasty used to prey on them in their native swamps, but that something either went extinct at a fortuitous point in Ferengi evolution, or instead the Ferengi only began to hold them at bay after learning to control fire, make spear points, and other Stone Age technology. And maybe only truly eliminated this other species as a threat as their technology progressed further into Bronze Age/Iron Age equivalents.