...As for the torpedo math, it's a particularly damning example of ill-thought-out technobabble in the original show. But it isn't all that difficult to explain away.
Of course, when Kirk wonders about the thing's ability to take a pounding, he already knows it is "very small", and later Spock clarifies that this means "a fraction over one meter". Assuming Kirk is fluent in Spockese and can deduce the latter from the former already, it does seem logical for him to think that a thing less
than a hundredth the size of Kirk's ship should be easily destroyed by a force more
than a hundredth (that is, one-ninetieth) the size of the force that Kirk's ship just withstood.
But how many photon torpedoes does it take to destroy a TOS starship? In the teaser to "Errand of Mercy", the Klingons pound the ship with half a dozen sucker punches of what very much look like photon torpedoes, and Spock reports "minor buckling" only. This would be quite consistent with a starship surviving several hundred hits before having to cry mommy - that is, consistent with Jim Kirk's starship being the equivalent of Horatio Hornblower's windjammer, rather than of a WWI battleship (a dozen good hits from contemporary main weapons will do) or a modern tin can warship (a single hit suffices). When Kirk in turn fires torpedoes at opponents, he doesn't give up because the first dozen would fail to have an effect; in "Journey to Babel", he gives up because he fails to score hits. So we don't learn much about the ability of starship-sized regular enemies to withstand photon torpedoes, either.
taking the shields down to 20%, how could the Enterprise expect to take another four hits
It's down by 20%, not down to 20%. The math works there.
The problem is with the later, movie or TNG era portrayals of photon torpedoes as somewhat more effective in the ship-killing role. It doesn't take a hundred direct hits to bring a later-era starship to her knees - it only takes half a dozen. Even a single good hit might reduce shields by 20 percent. Are 24th century torpedoes really a hundred times better than 23rd century ones?