He ditched the Ponds in order to save them, and then he faked his own death. Realizing what a huge mistake that was, he returned to the Ponds at the end of last year's Christmas Special. The Ponds were his family, probably the first family he's had in a very, very long time. You don't just throw that away.