Season 4 isn't terrible, it's just not as good as the previous years. The only year where I really started losing interest was 6. I can't even remember who the bad guys were in that anymore. 8 is another total blank slate for me but to be fair I only saw it a single time.
Season 4, 6 and 8 were the absolute worst seasons of 24. Season 4 gave us the Rube Goldberg Houdini villain who had a plan for a plan for a plan and could escape CTU perimeters without breaking a sweat. It also gave us the "daughter subplot" from hell and Edgar Stiles.