For me, it was the way he was used, and the way characters reacted to him. If he was written as an intelligent kid who reached higher than he should have, and occasionally got burned because of it, he would have been fine. It's that he was treated as a wunderkind (in-show and metatextually) that irritated me.
As others noted, his annoyance factor was greatly reduced when he became a cadet
It's what people who watched Smallville would call the 'Lana factor' - a character who can do no wrong in the minds of characters and writers despite being a pretty crappy creation.