He lives soooo long. He has to know and accept that his companions don't.
Except the Ponds were far more than his normal companions. They were his family. By the end of it, Amy and Rory had pretty much become his adopted parents (age differences notwithstanding), and then they were ripped
out of his life. I thought his angst was perfectly justified in this instance.