Doesn't invalidate the idea of Kirk visiting him.
Well, Khan's own words do:
KHAN to Chekov : This is Ceti Alpha Five. ...Ceti Alpha Six exploded six months after we were left here. The shock shifted the orbit of this planet and everything was laid waste. Admiral Kirk ...never bothered to check on our progress.
Khan makes it clear that Kirk never came back after dumping him on the planet.