When you write a prequel, you're pretty much writing something that only a fan of the existing material would have a natural interest in. I agree it wasn't a bad idea, but I do think a majority of viewers were confirmed fans, even if only casual at first.
I do know what you mean about going through phases of just not being as interested. I definitely felt like I was done with Trek when Enterprise ended, mainly because I'd been watching it for 15 years nonstop and it felt like way too much. Give me a chance to miss you, Star Trek. Well, it did, and I'm back.