It was The God Complex where they left, actually. But yeah, I agree in full, that was the best way to leave them. They're done with their TARDIS travels and are living happily ever after. Instead, in the end they're left stranded in the past to live out the rest of their lives, which we know they have since we saw their grave saying they lived into their 80s. Worse, there are probably a dozen ways the Doctor could have saved them but didn't with no reason given. True, they really were his in-laws. I can understand going into the kind of depression he did after losing them.