I believe Spock was speaking literally when he said Reliant could outrun and outgun Enterprise.
So,why did Enterprise win the day?
Ironically,in this fight Kirk had all the advantages.While Khan stole an ideal starship for his goals,he either failed to understand or ignored the sparcity of his crew.
Yes,Enterprise took a severe beating in the first act.However,trainees or not,Kirk had a full staff of knowledgeable starfleet personell to restore whatever was damaged.
Khan obviously did not.Once something gets shot out on Reliant,it was staying broken.Kirk could lose his engines and warp drive,and fix it all with enough time.When Khan lost his warp nacelle it was curtains-whats he gonna do,go to a fleet yard for a new one?
Genetically engineered or not,there's no way you're gonna learn how to operate a starship that complex-and fix it-in the short time Khan had under battlefield conditions.
Hence why a showdown in a Nebula was preferable to a fight in open space-one shot from Enterprise did more damage than 3 from the gun-toting Reliant.And huge engines don't matter when you're blind-where ya gonna run to?