It seems to me that the officer recruitment would be looking for potential rather than academic achievement. Since Nog was nothing if not extremely bright, this would have been recognised. I agree with everyone who said this was an excellent plot device, especially since he had both the skepticism of regular Starfleet plus the horror of his own family to contend with.
Of course it was started all the way back with Worf and reminds me a bit of Terry Pratchett's Watch, where the police force are the first to recruit all the types of people that nobody else had ever considered employable.
It doesn't really work like that in any real world way.
The U.S Navy etc WILL NOT take you into officer training without appropriate academic achievement, and if you have a criminal record you might as well just forget it. It doesn't matter how smart you are, what your IQ is etc, you need the qualifications.
A lot of people don't even get to the academy, and even if they do, passing it and then graduating to an officer position would be incredibly difficult, with many probably dropping out into NCO positions. Considering the Academy takes Vulcans and many other races, I don't think Nog would be anything special in the ability department.
It could be different in the star trek universe, but I put it down to lazy writing or "you weren't meant to think about it" rather than it being a logical thing.