I don't think it's the particulars that matter here in terms of if the Ligonians are wearing clothing that is more far east than Ivory Coast, etc. Hollywood has a long tradition of mixing up cultures and lands when portraying foreign countries. It's the aggregate impression
that matters here.
We'd probably not be having this discussion had the Ligonian costumes and language been portrayed as weirdly Victorian but with the same script. No, it's the combination of what what we see as stereotypical "tribal" clothing along with a "fight to the death", the all black cast and the kidnapping of the white woman plot that adds up to a racist image in the mind of many, and we can argue whether or nor that's what's really happening, but that so many people see it as racist certainly says that the choices made for the episode were probably the wrong ones.