I would guess that the farm can only support so many people plus the religious leanings of the group there.
They've got way
more farmland than they could ever handle with only one or two able-bodied women. I'm betting Hershel's got at least a couple thousand acres which could easily feed a few hundred people.
It's amazing Rick never mentioned that in arguing to stay. Farming without modern equipment sure as hell isn't going to be easy.