They brought in Trek novelists Judith and Garfield Reeves-Stevens in the final season, and the series takes a real turn from 'let's just do Star Trek stuff in the past where the ships move slower' to 'the first steps leading to the birth of the Federation'.Good heavens. It's almost enough to make me interested in ENT.
You could probably watch the Vulcan+Andorian episodes of S1-3 (Andorian Incident, Cease Fire etc.), then go straight into S4 and be satisfied with what you get. In fact I basically did that myself the first time I watched it, as I lost interest partway into season 1, and didn't really come back until the last third of s3.