Our image of the first Doctor is a bit after-the fact, down to descriptions in 70s books and Achellios' use of reference photos from Toymaker and Three Doctors for his covers.
Well, my image of the First Doctor is based on actually seeing him in the surviving serials. When my PBS station showed
Doctor Who starting in the '80s, it started with Tom Baker, but it eventually cycled back around to "An Unearthly Child," and they showed all the complete serials that survived from the Hartnell and Troughton eras. So I knew the First Doctor from, let's see, "An Unearthly Child," "The Daleks," "The Edge of Destruction," "The Keys of Marinus," "The Aztecs," "The Sensorites," "Planet of Giants," "The Dalek Invasion of Earth," "The Rescue," "The Romans," "The Web Planet," "The Space Museum," "The Chase," "The Time Meddler," "The Ark," "The Gunfighters," and "The War Machines," as well as "The Three Doctors" and the Hurndall version in "The Five Doctors." So really, I should've remembered that the Doctor could hold his own in a fight, but I didn't.
We had much less of Troughton, by the way -- all we got was "The Dominators," "The Mind Robber" (minus the short first episode), "The Krotons," "The Seeds of Death," and "The War Games." Although they added "The Tomb of the Cybermen" to the rotation once it was recovered, and I think they may have shown it out of sequence as a special the first time it came out.
I like to think that he wasn't entirely deaged when the Time Destructor was put into reverse in Masterplan 12, but that's obviously retconning, not the intention at the time.
There's also the life-force draining he was subjected to in "The Savages," not long before the end of his incarnation. That may have also contributed to his "body wearing a bit thin." (Although he was in fine form in "The War Machines" immediately afterward. That one's fun to watch, because Hartnell is on the top of his game for the first time in a while, and he seems exhilarated by it -- the actor, I mean, not just the character. It's a bit surprising that his health declined so quickly thereafter.)