There's one instance where McCoy needs to give him an injection so he tears the sleeve of Kirk's shirt, in full view of the rest of the bridge crew. Can you imagine him doing that to Uhura or Janice Rand ?
Perhaps he did and that's why he's left the service and looks like a hobo by the time TMP comes around. Perhaps he was drummed out for sexual harassment.
LOL - racist abuse seems more likely...
On the other hand I can imagine Spock as his disciplinary hearing:
Spock: "To suggest that a Vulcan would leer at a Yeoman is not logical. As a member of an emotional species Yeoman Rand must have imagined the insult. We only get the urge once every seven years and I'm not due for at least another year."
Areel Shaw (adjusting cleavage in embarassment): "Erm Mister Spock, my face is up here!"