Well, there are a number of things to consider that could've impaired the Voyager from cruising as such a speed:
Fuel consumption and the need to find places to refuel
Maintenance and propulsion downtime
Damage sustained from the Caretaker's Array
I think that energy was a constant shadow over their heads, because they rationed out the use of the Replicators and Holodeck use.
Even then, the Voyager still managed to make it home in 23 years in the alternate (or original) Endgame timeline, even without using a Transwarp Conduit.
Additionally, being a frontline starship of exploration, Voyager would have taken time to explore Phenomena Of The Week while slowly making her way home, for all of the reasons you mentioned as well as plot needs.
Replicator and holodeck use had to be rationed in order to play along with plot elements of the show, but I would think that even a split-second of warp core power would be enough to power those systems for some time. It's like turning off your air conditioner to improve your gas mileage...it might help just a little bit, but doesn't make that much of an impact in the long haul.