Did the script or the direction ever call for him to smile once?
It's not all just the actor.
Plus the fact that we had 3 seasons and 6 TOS movies, where we had a chance to see De Kelly's different range of emotions/acting. You can't jam all that in a single movie. Sure, De Kelly's McCoy was subdued at times, laughing at other times but I think he's best known for his arguments with Spock, yelling at Kirk and just plain old grumpiness.
Not to mention that McCoy just went through a bitter divorce, so naturally he's going to be bitching more than usual, especially when you throw all the events of the movie on top of someone that's already ticked off.
If you have one movie, which could mean the death of the Trek franchise if it fails, you may have to overplay some of the roles of the actors, at least from a writing perspective.
Hopefully future movies will see Urban show a different side of McCoy, the smiling, compassionate, sometimes humorous one.