Another reason the Vengeance's attack might have been more successful: Marcus knows the Enterprise's specs. He could have tuned his weapons to punch through her shields.
As for Nero/Narda, would Nero even need red-matter warheads/mines? Late 24th C. starship, off the shelf is probably going to be able to curb stomp a ship a century or so older--especially when he's waiting like a trap door spider to pounce on them. Seemed to me that once he lost the element of surprise he wasn't doing to hot anymore.