Season 3 is one long story arc. Archer gets passingly gritty for at least a few episodes. It's a compelling and interesting story, the producer changed and it's quite a differently paced show. The pedestrian qualities of season 1 and 2 disappear.
Season 4 has quite a few double and triple episodes, some of which delve into Trek history you will be familiar with from the other series. I consider Season 4 ENT to have some of the best Star Trek there is. It's even better looking than Season 3.
I was kicking myself after quitting ENT in the first two seasons after I started Season 3! I remember how exciting it was to finally watch those seasons, thinking "WOW this is such great NEW Star Trek!"
So, yes you should definitely watch it. Start with the last ep of Season 2. Even though I think you should watch it to be completist I also think you should watch it because it is so damn good.