Kirk's orders where to park at the edge of Klingon space and aim the torpedoes at the planet, that they could travel at warp themselves and go the rest of the way.
In the film they imply they're in dangerous territory, that Klingons do patrol that far out, but it would take hours, even a couple of days before they were found.
They were never meant to go anyway deep into Klingon space, I know Kirk changed his mind about bringing Khan back alive, but he still wanted the Enterprise kept mostly out of harms way, making sure the Klingons never detected any Starfleet presence.
So the Mudd ship could have been at Warp for a couple of hours all the way to the planet, then the same back taking up most of the distance.
Then as I said above, the much faster return could have been the Vengeance pulling the Enterprise along in her much faster more powerful warp field, overriding the Enterprise's.