Season 4 was too much too late, and it became the TOS-tie-in show rather than the show's original premise. It felt like a gimmick, and that's why so many people consider it "fanwank" rather than "continuity." Other episodes, like Daedalus, even blatantly ignored ENT continuity and illustrate that the show basically became TNG-lite, just like VOY did.
I admit that I grokked this the first time around. Compared to the first season that gave us the most unlikable protagonist I would ever see until I watched Secret Life of the American Teenage and season two which gave up episodes about T'Pol's Ponn Farr and motherfuckin' Porthos, then hell yeah season four's an improvement. Coto and company also started to not completely ignore Mayweather and Sato, so kudos for that.
But after a second viewing? Meh. It was one of Trek's most consistently decent seasons, but there wasn't a single break-out episode in the bunch.
This analogy I got from another site pretty much says it best:
Season Four was like a football team behind 0-56 going into the 4th Quarter, the team getting their ass kicked then scores three touchdowns after all their fans have left the stadium anyway and not even the refs and coaches are watching anymore.