Indeed, given that Humans have only had practical intercontinental travel for a few hundred years, I think it is entirely plausible that different kinds of intelligent species could have evolved on the same planet.
I admit finding it hard that many different kinds of civilizable life would evolve on a single planet; it seems improbable. But, anyone paying attention to biology the past two decades and the discovery of how bizarre extremophiles can get should be a little humble about proclaiming things impossible, and it is, after all, a huge universe out there.
(And the Star Trek universe is also one with abundant meddlesome entities of incredible power and unclear agendas, just in case something can't possibly happen on its own, and I would try not to be dogmatic in declaring this can't possibly happen on its own.)
It is impossible for a mammal to be more closely related to a reptile than to all other mammals. Unless evolution works in a completely different way on other planets (which is almost impossible), the different species of Xindi are not closely related. A Xindi primate and a Xindi insectoid would be about as closely related as you are to a praying Mantis.