My take is that the Klingons equate uniforms with military service as warriors (naturally).
Therefore, Klingon uniforms feature armor and medals and ranks and honor badges and weaponry and all that. The uniform of a warrior.
Conversely, Worf's Starfleet uniform has none of that, barely rank insignia and Starfleet emblem and that's it. That's it??? And oh so colorful, like pajamas. Not the uniform of a warrior, more like a "child's uniform" (no armor, no weaponry, no medals of glory or honor).
Plus I think the Klingons are also meaning that if Worf were too engage in a hand-to-hand battle while in a Starfleet uniform, they would have a political battle, instead of just a Klingon battle.