Where are they coming from? They're coming from everywhere! It's been at least a year and a half since the apocalypse, so these walkers would have had all that time to wander from almost anywhere in North America. If they've been walking continuously, some of them could be from California! And we know that walkers congregate slowly but never break up into smaller groups. They just wander in the direction of the last sound they heard, and keep walking that way until something new attracts their attention. So I'm not surprised that large numbers keep building up at the fence, and that next week's episode will show the largest herd yet.
I'm just surprised the Woodbury survivors haven't suggested the use of noisemakers and ditches to the prison group council. Or maybe they did, and the prison group rejected it? I can't see why they would, though. Those fences that looked so secure at the beginning of season three aren't looking so formidable now!