I think the only time where the continuity issues really annoyed me was "Stigma" when 35 years of previously established Trek canon was deliberately thrown over board in a massive retcon in order to come up with an - at best - average AIDS allegory. Fortunately, this abomination was later "de-retconned" in Season 4. Other things didn't bother me that much. For instance, just because the Xindi had never previously appeared in another Trek show it doesn't mean that they never attacked Earth in the 22nd century. There are a lot of species (and a lot of historical events) out there and you can't expect that they are constantly mentioned.