The explanation that it was a "refit" was a bit of a stretch, unless you assume "refit" means "Stripped down to the girders, then given an extensively redesigned spaceframe before being completely reskinned."
I do remember reading somewhere, maybe the novelization, that the ship was stripped down to the skeletal frame and computer core, and rebuilt from there up. So Decker's line that it was an almost completely new Enterprise was literally true.
Somewhere on this BBS a few years ago, someone posted a scan of a Starlog letter by someone insisting that it was impossible for ST:TMP and TWOK to be in the same reality as TOS because all the designs were so completely different. There are always some people who can't accept a change, even if most fans are fine with it.
That's interesting, and not really surprising as I've read comments from people here and back when I used to frequent trekmovie.com. One thing I've learned about Trekkies is we have our own IDIC. We Trekkies rarely agree on anything, except that we love Star Trek (or at least some form of Star Trek).
It's interesting for me as well because I come from the other direction. My first exposure to Star Trek was the movies. TSFS then TWOK. Then I became a Trekkie after seeing TMP (yeah, I sort of watched them to start in reverse order--but after watching TMP I went back and watched TWOK and TSFS in proper order

). In a way it made it easier to accept some changes. Now it helped that TMP offered some in-universe explanations for it's changes. They didn't just say 'let's pretend this is how the Enterprise always looked'. They acknowledged the change there. But since that was the first thing I saw, and I knew from TMP it was different when I went back to watch the TV series it wasn't an issue for me.
I went to see TMP in the theater this evening in fact (I was thrilled--TMP is the only Trek film I never saw in the theater and being my favorite Star Trek film I was glad to finally see it in all its theatrical glory) and during the pre-show interviews one of the associate producers said he believed TMP was the first movie to follow a TV series and be a sequel to that series with the same actors, characters and all. Nowadays I guess that's not such a big deal. But it's pretty cool to think Star Trek started as a TV series, continued as a movie series, then continued later as further TV series, movies based on one of the spin-offs, more TV series and movies and series again, and they all belong to the same basic universe (even the Kelvin verse movies--which are still tied to the original). So many franchises have reboots that basically ignore previous iterations. All of Star Trek is still tied, not just in name but even in story continuity, to those 3 seasons of the original series (and too each other even--Discovery acknowledges Enterprise for instance). That's pretty amazing if you think about it. How many franchises can say that--that they are still tied in-continuity--to what started it all.