First: there are no shockwaves in space, in the same sense as on Earth (in the atmosphere). Shock waves in space are purely particulate, which means the tiny little pieces of Vengeance that might have hit the Enterprise would have to have been enough to push it. Translation: not likely.
Yes, there are shockwaves in space. If you have an outburst of EM radiation and particles it has to go somewhere, whether there is air between you or not. There would be a lot more than tiny pieces bombarding the Enterprise as a result of the massive explosion. Your chance of avoiding getting hit is if the outer hull of the Vengeance absorbed most of the energy and went flying away in one piece. Dialogue would suggest otherwise.
It was established in TOS: Balance of Terror that you can use proximity blasts in Star Trek, and from there on space shockwaves have been quite common. Remember The Wrath of Khan or The Undiscovered Country?
ETT: The same story would have had better science in TOS for one reason – the budget would have not allowed for such a realistic shot of the Moon sitting next to the Enterprise, and you would have had no idea where the Enterprise was. Probably every TOS episode where the Enterprise had a close encounter with a planet made as much sense as that scene, they just didn't have to money to show it to you. Or do you really think they choreographed their orbital manoeuvres with precision back then?