^
I think it's more basic than that.
Like Picard said, the Q are really just glorified flimflam men. Because of their longevity and heighten senses, El-Aurians have the ability to see through the Q's tricks (or "behind the curtain," if you will) and thus the Q's power has little or no effect on them. The Q then feel threatened by this.