I don't think it was harsh of Marshall at all. He's always been the more considerate of the two, always supportive of Lily pursuing her dream. He's also probably right that if things had gone differently in San Francisco she might never have gone back to him. He's earned a little selfishness at this point, IMO.