It wasn't as successful as TNG because it wasn't as broad and was too complex for most viewers. Nothing to do with quality. If somebody tuned in to TNG for the first time during S6 they'd understand pretty much everything. If a new viewer tuned in to DS9 S6 they'd be utterly lost.
And that's not a bad thing. Not every show should aim to please everybody. Risks should be taken.