So what could have caused the first treaty to break down?
I'm not sure it even broke down. I'm sure nobody thought that the Khitomer Accord was going to instantaneously create a new era of perfect peace and cooperation, in which the Federation and the Empire always get along, but only that it marked the first step in a long, rocky journey to toward better relations between two long-time adversaries. There were bound to be plenty of bumps and setbacks along the way.
I mean, look at the USA and Russia. The Cold War ended roughly two decades ago, but it's not as though America and Russian are in perfect harmony these days!