I've watched that episode a lot over 50 years, and I don't recall Balok announcing that the engine failure on his ship was faked.
"And I thought my distress signal quite clever. It was a pleasure testing you." That, plus the fact that the life support system is working perfectly when the distress call said it had failed, and that Balok acts as though everything is fine rather than asking for help repairing his engines, makes it pretty clear that the distress was faked to see how the humans would react. It was the final stage of the test to see if they were as benevolent as they claimed. "I had to discover your real intentions. ...Your records could have been a deception on your part."
Clearly this is an unusual defence strategy. They can't build a huge intimidating space fleet, so they spend what they have on a great big scary watchdog with no teeth. They hope to scare potential invaders away. If it's ALL faked, if there's no First Federation, any understandable motivation for what he does vanishes instantly.
That's how I've always seen it, and how I approached it in The Face of the Unknown, the Definitive First Federation Novel I've wanted to write my whole career and finally got to write a couple of years ago.
Although I don't agree with the "no teeth" part. The Fesarius was obviously quite powerful. If it had the power to hold the Enterprise in a tractor beam and require maximum engine thrust to escape, then it could easily have directed that same power to weapons and destroyed the ship. And the cube buoy emitted enough radiation to kill the crew if it hadn't been destroyed first. It's not that Balok couldn't have destroyed the Enterprise, it's that he didn't want to -- at least, not unless they'd failed his tests and turned out to be hostile.