Silence will fall when the Question is asked. Well, the Question was asked. What exactly was supposed to happen? Why was Madame Kovarian so crabby?
Well, we've seen what was supposed to happen, haven't we? Billions of people dead, history rewriting itself. The Silents were just trying to prevent all that from happening, they were the good guys! Hell, I'm sure one of them works with Madame Vastra and we've just forgotten about him!
Also, it must be said that the short scene in which the Doctor was crying was heartbreaking and very well acted.