In warships size MATTERS.
More power, heavier weaponry.
There's no such thing as a magical "fit everything that makes a cruiser into a destroyer".
Plus, Voyager was shown as having an equally large percentage of amenities as the Enterprise (crew quarters, etc). It's not even built with a Defiant-like philosophy.
At an equal technological level (which is the case), the larger ships wins, easily.
Meaning - Voyager gets fried against the Enterprise.