A lot of good accounts of the series here. From what I remember reading on here at the time, the main reason was people expected to see the series progress towards the forming of the Federation. Seasons 1-2 felt like generic exploration. While the Andorians appeared early (people seemed most happy with the Andorians' development in the series, even in Seasons 1-2), the series seemed to lack focus and wasn't showing enough classic races. Episodes felt like, with a few script changes, they could've been TNG or VOY episodes. I think there were far too many unknown races being encountered in close space, making people feel this was several hundreds of ly away from Earth, not within 100, 200, etc ly of Earth. Add in to that scripts many felt were bland or were patronizing (Ferengi doing the Rumplestiltskin routine) or were ripoffs ("Oasis", "Doctor's Orders" are the biggest offenders). I think, even with the lack of creativity with the technology or looking more high tech than TOS ever did, would have been overlooked by these people if they felt Seasons 1-2 had a direction and could see steps being taken towards the founding of the Federation. There were too few of these episodes. Also, if B&B had openly addressed the Klingon & Vulcan matter, saying it was part of the plot and people would see them take their classic forms over the series, I think enough people would have given them some leeway.
A secondary reason is the Temporal Cold War. People seemed intrigued by it, supported it, but that all changed in Season 2. People seemed to realize B&B were making it up on the fly and didn't have anything plotted out for it. I think that's where X-Files influences things. X-Files ran from 1993-2002, they had their movie in 1998, killed the Syndicate in 1999, and eventually launched a new mytharc in 2000 (a far lamer one with supersoldiers). X-Files and its mytharc had many fans until people realized these weren't all pieces in the same puzzle they were shown, they were random and there would be no payoff for carefully following the whole story. Carter & Spotnitz were making things up as they went along and weren't keeping track of the elements they were introducing. By the time ENT premiered, X-Files' 'genius' of ongoing story arcs was exposed, the curtain was pulled back. I think people didn't want that again. Not sure how many Enterprise viewers watched X-Files, but X-Files was hugely popular for a fair chunk of its run and was a breakout hit for 1990s Fox.
Ok, my take:
I watched its original run. It was acceptable even though I lacked the enthusiasm for seeing new episodes as I did for TNG, DS9, VOY (except for the Xindi arc). There was a stretch in Season 2 which was the only time I ever tuned out of Star Trek after returning to DS9 (hey, I was a kid then and Season 1 was soooo boring). I saw every new episode for years (except fell asleep during 1 Voyager episode and had to mute part of "Ferengi Love Songs"). Upon revisiting some of the series' episodes just in the past year, I felt ENT held up the least of all the series. DS9 was perhaps the most fresh because in some aspects it was ahead of its time in storytelling, subjects, but TNG and VOY didn't look dated and aged to me and TOS, while parts looked very 60s (it can never get away from that), the stories and everything still seemed to hold up. ENT just came off as very bland, even the characters and despite being the newest series, meaning it should have held up the most in time, it just came off as bland, forgettable, dated. I realized I was only watching it just because it was new Trek when no other Trek was on, not for intrinsically liking it. Watching it next to TOS episodes gives a sense of what it should have been. The uniforms were good though (invoked astronauts, flight suits more than later uniforms) and some of the attitude was (these were pioneers, the only ones out there. TOS had to recruit a lot of people to fill Starfleet, so you're going down on quality a bit there and there was a cavalier, Wild West mentality, not the 1960s space program kind of feel)- basically, think Lewis & Clark vs. the population that settled out there and acted out the Wild West over decades. That's the difference. Besides, even a modern TOS knockoff might've done better. It certainly would've been more in character with UPN's attitude and audience.
Season 3 I remember people were intrigued by the new elements (Xindi, Delphic Expanse, Spheres) and it did feel like a completely fresh break. Even though it ended up only being a tangent on the TCW, people seemed to like the story arc overall. The ratings were nearly flat for the season, suggesting high viewer retention (but not growth). People also saw the 9/11, Iraq War parallels and felt the light, irrelevant Seasons 1-2 could be replaced with something quite relevant and heavy-hitting.
Season 4 I remember was what many people wanted, but I think most knew the series was doomed then (Friday night death slot, was barely renewed at the end of Season 3, word going around it lost its supporters among Paramount brass). I remember it seemed like too much of a good thing, being too heavy on pleasing the fanbase, trying to emulate TOS stories (a 3-part Khan supermen arc, "Observer Effect") and was too arc-heavy (one arc to another to another). I think the ideal balance would have been a blending of the Season 1/2 model (exploration, many standalone eps) and Season 4 (small arcs, 2-parters interspersed which are all tied to building the Federation or developing the TOS-TNG Trek universe). It tried to make up lost ground, but felt like rushing to add everything in before the end (and too much fanservice feels patronizing).
I think a stretch in Season 2 just destroyed Archer's character, which was already getting a reputation as a doormat. I think they got close to the right idea (an explorer first, someone who tries to smooth things over and overcome differences, who can use words to calm tensions and bring people together) but Archer ended up being more hated than Janeway. I think "A Night at Sickbay" and "Singularity" (even though under the influence, I think it plugged into the stereotype for many. I think several Season 1 DS9 episodes gave people a negative stereotype of Bajorans, which helped alienate some viewers from that series) in particular seemed to take away any credibility he had for many people.