I think the main culprit was the "religion" aspect.
Bajorans on a whole were not interesting to casual fans imo, "Prophets, Pah-Wraiths" I thought DS9 spent a lot of time on them to no avail. As a hardcore fan, I can find that part of DS9 interesting, but I could see the casual TNG fan who maybe watches an episode if he has time would be turned off by that.
I thought DS9 got a lot right, the action scenes and core cast were it's strongest parts, and it knew how to tell a story better than any other Trek in my opinion. Had it dropped the religious undertones and been a little more traditional in that aspect I think it might have had a better interest amongst the casuals.
But all of that is just my opinion.