If babies weighed down humans, then our species wouldn't have advanced. For tens of thousands of years, our species was nomadic.
My understanding of the comics is that Rick's people moved up the coast to the environs around Washington, DC. I don't know if they will be doing that in the show. I do know that they are filming the first episodes of the fourth season in Georgia.
Are there advantages in moving to less populated states, where the population wasn't that big to start with, other than the one I enumerated? I am not sure how much of an advantage a group could gain from moving into a heavily populated area. Other the issue of encountering a large number of walkers, they would be fighting with other human groups for increasingly limited resources.