Given how little time passes between the crippling of Reliant and Khan's irreversible triggering of Genesis (it's the first thing we see him do), I think if Kirk had sent a team to Reliant by shuttle the only result would have been a dead team.
Nevermind that Reliant might still have possessed the ability to shoot down any incoming shuttle.
Which raises a good point: evidently transporters were still functional, Our Heroes did know that Reliant's weapons capability was moot, or Kirk was bluffing when he told Reliant to prepare to be boarded.
Yeah but as it's been mentioned a few times on Star Trek, and shown many times in reality, part of the responsibility of command is ordering men and/or women in to situations that have a high likelyhood of them dying. Sure Khan activated the device pretty quickly and a boarding team probably couldn't have stopped it but Kirk had no way of knowing if Khan was alive or not and regardless priority#1 should have been getting Genesis secured as fast as humanly possible just because of the sheer magnitude of the weapon. Maybe Khan was dead, maybe he was knocked out, maybe he was alive but couldn't move or he could only move so slowly it would have taken him 10 minutes to crawl over to the control panel and activate instead of the 1 or 2 it did.
The point is when something that powerful is not under your control, priority #1 is to get it back in your control and there is ZERO time to mess around. If Khan activates the device before they get there and the Enterprise has to flee and leave the team behind, well it was pretty much assumed that everyone was going to die regardless because noone knew Spock would sacrifice himself to save the Enterprise, so that's a non issue. If the Reliant has a functioning phaser and blows the shuttlecraft away.....well that sucks but it's a risk that they had to take. Everytime a flight of B-17's lifted off to bomb Germany the ground commanders knew some of them wouldn't make it back and men would die, but the importance of defeating the Third Reich took precedence.
To put it in real life perspective say there was a terrorist group that had a nuke in a wearhouse in NYC. And somehow the military was able to strike from outside in a way where they assumed everyone was dead or incapacitated. They aren't going to sit around with a bullhorn radioing instructions to surrender and come up with your hands up. They are going to send in a recovery team the instant they can, if some guy sets off the bomb or they get mowed down by a survivor with an AK47 before they get there......well that pretty much sucks but there are no guarantees in life, but the stakes are too high NOT to respond immediately.