I'm not sure about outside of TOS. But I can only find a reference to "Mister Spock is much stronger than the ordinary human being." Which is from "This Side of Paradise". As far as Klingons go, I don't think they ever gave any indication of their strength in comparison to humans.
In ST, humans are apparently the Federation Starships of the living world. That is, no matter what alien they go up against, and no matter the size difference, they get the crap kicked out of them.
It's amazing we made it out of the Stone Age....