It doesn't bother me that Prime said to kill them all, as out of character as it may seem. By this point, the Decepticons had tricked everyone into sending the Autobots away, and instantly capitalized on it to level an entire city, and slaughtering people just for fun. Optimus Prime realized there was no compassion or mercy among the Decepticons, and to protect humanity, decided he needed to be relentless.
Of course that's not how it really played out in the movie. It just comes across as yet another out of character moment, such as yelling "GIVE ME YOUR FACE", or hunting down and killing Decepticons who were apparently just hiding. But if he had 3 movies of being in character, and they played it that way, it could have worked very effectively. It just didn't.