I haven't read many TOS novels but I've been adding books to my list to read and Faces of Fire by Michael Jan Friedman has a movie era Klingon with 5yr mission Kirk on the front cover - could be this one?
No -- Kirk is actually on a different planet when the Klingons raid the colony and doesn't meet them face-to-face. And there's no mention in the book of anyone finding the Klingons' appearance unusual for the species.
However, a later Friedman novel, My Brother's Keeper: Enterprise
, shows Kirk encountering a genetically engineered breed of Klingons who are more savage than the norm, and it's implied (though I don't think it's stated outright) that this was the origin of the ridged Klingons (since it came out before ENT established the official origin). This might be what the OP is thinking of.