Lots of reasons. A couple of the most obvious:
Matt Jefferies never imagined fans would obsessively try to reverse-engineer full floor plans for the Enterprise when he designed it in the 1960's. His priorities where first and foremost to create something that would look interesting and memorable on TV screens, and secondly something that would be reasonably easy to build a model for. Everything else - including the overall size of the ship - was at best a reasonable, good-enough guess.
The show's budget was no doubt a significant factor in the relatively small interior sets. This had a lucky side-effect of assigning them some "realism" for those who imagined the Enterprise as something akin to a mid-20th-centry warship in space. I suspect this was another case of a tight budget actually being a blessing in disguise, similar to the well-known case of the invention of the transporters.
Many model builders have noted that you can't really put even those small sets as presented in TOS in the Enterprise unless you increase the size of the ship by about 42%. So you've already got to mess with un/official cannon (947 feet) even if you're keeping things the same but trying to be more realistic. See here for a recent discussion and example:
https://www.trekbbs.com/threads/new...1-foot-version-full-interior-3d-model.311118/
With that out of the way: Why was the Enterprise D made so much bigger? Surely, starship technology could've advanced greatly without the need to increase the overall size of the ship - or at least not the individual rooms/sets. And yet that's what we get: every set is larger; from the bridge to individual quarters. The answer is simply that they had a bigger budget and wanted to design bigger, more impressive sets for the now much savvier audiences. And they had hindsight to fall back on: they knew fans would study every inch of this thing, and they wanted things to match up. So... bigger, fancier sets = bigger ship.
That same desire/need for bigger, fancier, more-impressive-to-modern-audiences sets obviously holds true today. Only the most die-hard fans would seriously consider going back to the tiny, plain sets of the 1960's. So, if you're going to have to have sets at least around the size of what they did for TNG (if not bigger and fancier), then you're going to have to have a ship to match that.
Ergo... bigger Enterprise.
Blame it on the unimaginably wild success and longevity of a TV show no one seriously expected to still matter after the 1960's.
All that said: I do think they went too far with the up-sizing in the Kelvin universe. That was just... unnecessary... (And also, Discovery's bridge being largely empty space really bugs me.)