It's a variety of factors.
-Novelty of the concept wears off
-Writers often have a certain idea for the show, but after 4 years, they've gone as far as they can without reinventing things, which often involves a change of direction
-Flip side of that is just continuing to go with the formula, which may be fine for some, but others drop out as the show becomes too predictible.
-Around the point where actors' contracts may start to run out, leading to cast changes, main characters leaving/dying/recast. Or if the show got big, actor may be a big deal now and want out to go after a movie career.
-Really think it's mostly just concept. You start with a great idea for a show, and unless you very carefully plan it out for a long duration run (and hard to know that in advance, so you often give a lot early in case it's shorter than you wanted), but after 3-4 years, you've usually exhausted most of what you set out for originally, and start stalling for time (see LOST, which is an easy example, and didn't pick back up until they had a firm end date, and could plan the rest out, or at least as much as they did, anyway
) Show may still have good storylines in it, or characters may be popular, but it's not usually as good as it was when the writers still had big plans for the show. Becomes a game of running out the clock, seeing how long you can keep the money flowing without having to start over...