I don't think it was harsh of Marshall at all. He's always been the more considerate of the two, always supportive of Lily pursuing her dream. He's also probably right that if things had gone differently in San Francisco she might never have gone back to him. He's earned a little selfishness at this point, IMO.

One thing I was disappointed about though was that we didn't get the Mother telling the story to her kids for this episode.