Okay, my honest opinion...
Seasons 1 and 2 are the same old, same old. Most of the time, it didn't really matter that this was a prequel, the stories could easily have been adapted to work on TNG or Voyager. Still, I like it - those first two seasons were solid (yet not exceptional) Trek. I do believe, though, that this is where the audience started leaving.
Season 3 was big, bold and fun. Great storytelling. But again, not much of a prequel - rather a reaction to 9/11. While I enjoyed it immensely, I was wondering when the show was going to live up to its promise of working towards the Federation we know and love from the 23rd and 24th century shows/movies. Many of those people still watching thought the same thing and tuned out.
I stuck with it, and season 4 is where we got the meat and taters. This is the show Enterprise should have been from day 1! Just perfect. Well, the budget constraints were noticeable in the fourth season - though this had both good effects (two- and three-part episodes, so that sets could be used for more than one show) and bad (synth soundtrack). The writing was top notch. If it had been like this from the start, it could have gone for seven years. (Well, there was still the oversaturation viewers might have felt after fourteen years and 526 episodes of 24th century Trek.)