KRAD's IKS Gorkon books are practically required reading on this subject.
You can't just challenge someone to combat without good cause, and certainly not someone of high rank or social status. If that person's bodyguards (or even just the onlookers) see that the challenge isn't valid, they'll kill the challenger for his impertinence. As has been mentioned, Klingon society simply has to have evolved these checks and balances, to prevent total chaos.
I read the books but i completly forgot about this.
Nice catch, both of you