After watching the Talking Dead here is what I may have noticed:
- no walkers in the field at the prison or along the fence
- no signs of life at the prison at all
I'm of two minds about this:
a). Rick and Crew left the prison...
b). they left the prison waited outside woodbury for The Governor and his men to leave and then took over Woodbury.
Perhaps they moved the walkers inside the prison o when they open the door to get into the prison the walkers come out and attack the governers men.