Because of the three primary characters in TOS--Kirk, Spock, and McCoy--we learned what they were up to during that time. Kirk became Chief of Starfleet Operations, Spock left Starfleet to undergo the Kolinahr, and McCoy quit Starfleet too to audition for the Village People. Don't know about you, but that was enough for me.Why?
Now, we could say all sorts of other stuff happened between TOS and TMP--maybe there was a second five-year mission, maybe Kirk was the planetary governor of Deneva for a time, Spock and Uhura were briefly married then divorced, and McCoy became the Karaoke Champion of the Federation. Can't preclude those possibilities too by that logic.