I don't think it will just be the Governor and 2 men. I think they will join forces with other devolving survivors.
When they were still on the farm the kid they had tied in the barn (I forget his name right now) said he was part of a group of about 30 men who had very violent methods. They would know about the residents of the farm and would want their own form of revenge.
The graphic novels depict several different groups of survivors. These 3 won't be on their own for long.