I go for the plural option. The urge to say "by god" or such is there, but I've successfully trained myself to morph it into "by the gods", etc. To use the singular suggests the Christian/Islamic/Jewish deity, and that might:
a) suggest my affiliation with one of those religions, which being non-religious I naturally wish to avoid.
b) offend those who do believe in that god (note I'm not one of those "must walk on eggshells, mustn't be offensive!" types; I simply don't see the need to risk offense when it's no trouble to me to avoid it).
Saying "the gods" makes it a general invocation of humanity's gods and the concept of deitys and such, without affiliation or taking any particular deity in vain.