I've never heard anyone say "only" and if I did hear that I would assume they are minimizing the other stuff as a way of highlighting the importance of their relationship to their children. I think it's just a turn of phrase. It's easy to look at all the dribble life gets filled up with and say, "the only thing real is the kids".