Talking Dead had a good short discussion on how Rick's moment of craziness drove Tyrese et al to the Governor.
And Martinez is a big threat. Not to the Governor, at least not yet, because he doesn't seem to want to be boss, but because he likes to bully.
Merle was fitting in pretty well there. Surprise! But maybe that's become his nature, after seeing how crazy the Governor is.
And I don't want them to kill Judith. I want them to get the prison cleaned out a bit more. Settle there a bit. Though the show is built on a nomadic existence, that nowhere is safe anymore. I mean, how will you have "the walking dead" be a threat if they're all safely outside the fence? The focus of the show would have to change. If they stay at the prison, rebuild it, expand a bit--the next threat could be from any leftover governmental types.