The biters in Season 1 were freshly dead, and still had muscles and some "brain power" (remnant imagination, perhaps?) to motivate them to do things like climb fences and smash glass with rocks. By Season 3, after almost an entire year, the biters are significantly decayed (as shown by the makeup) and are probably lacking in strength due to decayed muscles and any "imagination" they have is gone. They just walk until they bump into something.
Furthermore, the biters are only really motivated to act when they sense prey by sight or smell, and most of the time Rick's group isn't hanging out by the fence to antagonize them. So they just ceaselessly bump against the fence. If one walker did show any inclination to scale the prison fence, someone would merely need to destroy that one before they made it to the top.
To really invade the prison, a large herd would be needed to just push the fences down with their mass, as happened on Hershel's farm and in "Guts". That could happen someday. Perhaps the Governor could round up a large herd and aim them at the prison.