Well, The TARDIS did fall at/to Trenzalore with 11 in it.
Silence did fall when the question was asked and The Doctor didn't say anything at first
Maybe the Silence were involved in a battle that the Doctor helped prevent or stop? Him being undone by the GI would mean that they would've been defeated or completely eradicated. So him answering the question, and allowing the GI to do what he did, would result in their fall.