^^ That's quite a can worm opener you brought along and a delicate subject.
Indeed, if I'm supposed to take it literally that our Kirk visited an antimatter universe in "The Alternative Factor" or that dilithium crystals (matter) are supposedly immune to antimatter, I'd rather assume "antimatter" to be some form of colloquialism as you just suggested.
On the other hand it's pretty much spoken out, that matter-antimatter annihilation somehow
propels the Enterprise
The question would remain what kind of colloquialism "matter" would stand for.