The one thing speaking against that is that Uhura was sending messages of "Prepare to be boarded" for several minutes before the subject of "You can't" arose. Why did Spock say nothing? Was he thinking that Kirk's parties could board the Reliant
via shuttles, spacewalks or hard docking even though the transporters were out of the question? Or that Kirk was bluffing for some reason and it was not Spock's place to point out the weaknesses of the bluff?
Mind you, it would be advantageous to send Khan the standard Starfleet surrender ultimatum even if there were practical problems with it. Kirk should press his advantage, and asking the impossible from the defeated party is good psychology that asserts Kirk's dominance.