Technically it's an accurate term. Kheylar was the only Klingon(or part Klingon) he was ever with, and she refused the whole oath thing only to have Worf refuse it once she got over her cold feet. The only reason Worf even bothered acknowledging that they had mated was so he could kill Duras, otherwise he wouldn't have.
So if not girlfriend, what else would you call her? Friend with benefits? Enemy with benefits?