The groups long term survival depends on them being in the prison. They barely survived last winter and now they have a baby weighing them down. The prison is essentially a modern day keep. All they have to do is find a save vantage point and shoot to kill when the woodburies show up. Woodbury may out number them, but the vast majority of the population is soft city folk who will be totally out of their element.