They also can't apparently use their power if they're under any kind of physical duress. You could probably just slap one and they'd be like "OMG WTF Y U HIT ME?!" and then have to start powering it up again.
The only reason the one in the restroom took out the older woman was because she was completely unaware of the threat and Amy was too shocked to do anything about it (and not under any subliminal order to kill on sight).
So unless a Silent was making a preemptive strike from behind, chances are just about anyone could survive their 'attack.'
Which kind of makes sense. If no one can remember you, why would you need a hardcore offensive ability let alone any other kind of defense mechanism? Especially if you've been under that for hundreds of thousands of years (remember, the Silent said he was there when we were learning to use fire and the wheel).
But, again, I still don't understand why the Doctor found them to be a threat to Humanity. Clearly if what was said was true and the Silence were responsible for our entire development as a culture, Earth would instead be a symbiotic world of both the Silence and the Humans. And other than killing the older woman in the restroom for no apparent reason (I still don't understand what happened there), what 'great evil' did the Silence do beyond trying to protect themselves from discovery? Especially to the point of the Doctor -- the Doctor -- condoning and bringing about an act of genocide?
Of course the real question is: Why didn't the Silence just orchestrate a counter subliminal message? They could implant it into all kinds of video and radio signals. Heck, just make it so it gets transmitted every five minutes, and make it a message that counteracts the order for genocide.