Mental stress can have physical side effects. Increase heart rate, higher blood pressure, it can throw your chemical balances all off. That's what I wager McCoy was talking about more than anything.
Exactly. McCoy's exact words were "Jim, your vitals are way off." Presumably he means vitals in the same way today's medical professionals do: Heart rate, blood pressure, pulse ox, presumably because of the stress he's under. It was just a quick dialogue device to further convey to the audience that shit ain't right
, like Spock's objections to the mission and the inversion of the theme that plays when the shuttle is leaving for the Enterprise.