Probably. Because every character has their own set of vocal fans.
We probably would have had some people accusing Voyager of being racist as well, since Wang is Asian. Which would be complete garbage of course, but some people would say it.
I don't think it is garbage at all.
Mulgrew, Picardo, McNiell, Philips, Lien & Ryan are all White actors on the show. They out number the miniorities 5 to 1. Wang is the first Asian actor to be in the main cast, Takei doesn't even count because anyone outside of Shatner, Kelley & Nimoy were all supporting cast on TOS until the films.(It's why Takei's bitching about Shatner stealing all their screne time has no merit) Losing Wang would have been a step back and hypocritical towards Treks theme of diversity, Lien would not.