On some level I agree with many of the sentiments expressed in this thread -- that DS9 was "ahead of its time"; that audiences in the mid-90s probably reacted against it worse than audiences today probably would -- but it can't be denied that, for whatever reason, DS9 was less popular than TNG. I think this is just the luck of the draw though. The trouble with spin-offs is that risk of decreasing returns. Its possible that any follow-up to the massive success of TNG would have been given a (relatively) lukewarm reception.
Personally, I appreciated DS9 at the time -- heck, I appreciated VOYAGER at the time too -- but I can also see why there were some quarters with who DS9 just didn't "take".
Well as you say there is the rule of dimminishing returns. Which would explain VOY and ENT a bit, but there is more to it than that. For some who where new to the franchise might find those shows interesting, but for other longer term followers of the franchise they might have found those shows dull and repatitive of what the previous shows has done before. TNG's biggest advantage when it came out was it was one of the few Sci-Fi genre shows on at that time. It's success demonstrated that Sci-Fi could work on TV and everyone elsershed to get Sci-Fi shows out so by the time DSN came out there was far more competition for the same audiance group. If two networks schedule two Sci-Fi shows opposite each other one of them is going to lose out. Rememer this is the days before DVR's, and we where still on VHS, so we as an audiance might not miss it, but the ratings wouldn't show it.
Today, I can DVR weeks worth of episodes to hit in a marathon session. With today's 24/7 lifestyle, it's not always easy to sit down when something first airs on TV. TV has changed a lot over the decades, and audiance figures have been in general decline for years. Is this because of comptition or quality or a combination of both?