The prison is not secured. This was established in the second episode with the prison. The side with the administration building is wrecked. The "tombs" are overrun by zombies.
I saw the truck Glenn was driving, and I was like, "Whoa, that was one nice looking truck. Look at how clean and shiny it is."
I don't think the Grimes Clan is capable of taking on Woodbury. They are suffering from borderline starvation, are low on ammunition, and have fewer numbers than the Governors' Clan.