It has been stated by the producers that all Air Nomads were airbenders. They were unique in that respect. Well, yeah, but would nomads have villages at all? More like temporary encampments, or way stations that they periodically returned to as part of a seasonal migration (though I always figured that's what the air temples were and why there were four of them). The Air Nomads were always the smallest of the four populations anyway, and Sozin and his heirs had a hundred years to hunt down any stragglers no matter where they fled.