A nebula that has been contracting for ages, to where it's just about to start forming a new solar system, would surely be dense as hell. And that's what I think Mutara is. It's thick and swirly, hard to see through, and its particles get in the way of your phaser beams, reducing their effectiveness.
It'll still not be dense enough to do jack-all.
Space is very big, there may be a core around the proto-star(s), but the upper range is around a measly 10,000+ particles per cm^3, or, for a cubic kilometer, times that by around 100,000,000,000.... 1,000,000,000,000,000 particles, with temperatures colder than Pluto. The Reliant and Enterprise would be glaring hot and able to see each other nearly no matter where they would be in a nebula.
For comparison, A mere cubic meter on Earth has around 10 trillion trillion molecules, and a cubic KM much more so, and we're not generally in swirling purple mists and can't detect heat or see anything past a few feet, unless a fog rolls in.
Nebulas do not work like they're shown to in Trek, period. (or nearly any scifi).
Whatever the Enterprise and Reliant was fighting in, it wasn't a nebula. Something that thick with those properties would be more befitting for a gas giant's atmosphere.