Prejudice against the Klingons doesn't really stem from racism at all. It stems from the fact they have a culture that celebrates violence, that has a proven track record of consistent aggression against humanity.
Racism would be irrational hatred based on their appearance or whatever. The hatred of Klingon culture is entirely rational. If it existed, I'd hate it too. Kirk doesn't distrust the Klingons because they look funny. He distrusts them because their entire culture revolves around killing people for fun.