I think the Ferengi ended up being a cool creation in the end (mostly thanks to Quark), but yeah, as a villain they were nearly impossible to take seriously.
Maybe if they had been larger and more imposing the idea could have worked. But it seemed like the producers were trying so hard to make them "strange and alien" that they forgot that they still needed to be, you know... intimidating.
Agreed, DS9 did
go a long way toward fleshing out the Ferengi and making them a bit more credible Trek alien species but hardly makes up for the grandness they set-up to be in early-TNG. And well, even on DS9 pretty much any episode that focused solely on the Ferengi or around the Grand Nagus (with all respect, admiration, and love to Wallice Shawn) was OtT absurd.