Yet a cat and a serval can successfully interbreed...they're actually not of the same genus, but still the family. Let me introduce you to the adorable (but very pricey) result:
(If you're not familiar with the Savannah breed, F1 indicates that this cat is a first-generation mix, so 50% housecat, 50% serval.)
Now, IRL I totally agree that you would not see that kind of interbreeding with the kinds of species we see on Trek, excepting certain cases where the "same genus" thing is VERY obvious.
But, while the science is certainly not what would fly in the real world, what I'm trying to point out is that if you approach the problem through storytelling logic instead of through scientific logic, there is sense in the writers' decisions.