I imagine they used a system similar to the Navy.
Here's a link for more details;
Also there was a TNG episode that mentions this. I don't know all the episode names like you guys but it was the one where Cpt Picard goes off on a mission and the Enterprise gets a new Captain. The new Captain changes the duty rotation and Riker complains to him about it going from an 8 hour shift to a 12 hour shift.
Yes I thought they used a 12 hour watch while on a 'mission' and an 8 hour otherwise. While on something important like 'Balance of Terror' I think the main crew were probably on watch most of the time with perhaps a 6 hour sleep shift when they could. Kirk needed his main team on duty when things got critical.
I don't think current navy ships are directly comparable to a Starship crew. With no disrespect intended I think the Starship crew would have a lot more scientists, engineers, doctors, nurses than say a regular Navy ship. A lot of tedious tasks like swabbing the deck would be automated so a 12-hour shift probably isn't necessary for a Starship crew normally.
Also they're on a 5-year mission. They'd have to have some leisure time. Hence the luxurious boudoirs and recrooms in TNG.