Yet for some reason, the Romulans are definitely considered a threat by the Federation in TNG.
They feared going to war with the Romulans.
From the star charts I've seen, the Federation doesn't exactly dwarf the Romulan Empire, but is just a bit larger. (I don't know if these charts can be considered canon, though.
I don't think the idea is too strange, since history has examples of smaller sized powers conquering and defeating other powers with larger territories.