I completely agree with this. We saw adventures they had in the episode but there was time inbetween that, days even weeks. I'm sure much of their time was spent just quietly flying in a straight line. Not much to do but build shuttles. This is why I've always thought that even if they were able to use the holodeck energy that they shouldn't use it for anything but the holodecks. They will need the recreation for their own sanity.
Well we don't know what else they did for fun, though I'm sure they did things like read, listen to music etc. Sure there would be the odd bit of shore leave to help. But it would be around 2 hours of holodeck time a week for a crew of around 15 with 2 holodecks.