It wasn't as successful as TNG because it wasn't as broad and was too complex for most viewers. Nothing to do with quality. If somebody tuned in to TNG for the first time during S6 they'd understand pretty much everything. If a new viewer tuned in to DS9 S6 they'd be utterly lost.
And that's not a bad thing. Not every show should aim to please everybody. Risks should be taken.
Well said. I'll say, I was a kid when TNG came out, and only slightly older when DS9 came on. TNG was on at 8 or 9 PM, DS9 wasn't on until 11:00 in Kansas City. Also, as an 8 year old, it's a heck of a lot easier to follow TNG than DS9, it was the cultural phenomenon, DS9 was a "spin-off". If both had aired in the age of DVR and internet viewing, DS9 would have been adored by the masses.