Durin' the Occupation Arc, Damar was a heavy drinker (not yet a drunk, but gettin' there) and Kira was able to take him by surprise.
Hell, I once took on a drunk Army Ranger durin' a routine part of my job, and was winnin' 'til he rammed me into the hallway wall. So, its not just trainin' or physical prowess, tactics helped too.
As for Damar still carryin' the hate for Kira...like he said, it was a luxury he couldn't afford. He had to let it go to keep his mission goin'.