Yeah a non-political work still gets into murky waters when you have all the good guys as white and anyone with a darker skintone works for the Lord of Evil.
The superior race in The Lord of the Rings
and The Silmarillion
are clearly the Elves. In The Hobbit
, there isn't even a human as a main character. If the legendarium is racist, it's against humans in general, as they're shown as weaker and more corruptible than some other races.
Both Tolkien and Martin are writing stories focusing on medieval Europe-influenced societies, therefore most of the people will be white. To accuse them of racism for not bending the knee to political correctness and making sure they represent all races as having an arbitrarily equal amount of goodies and baddies is absurd. Thinking like that just leads to the death of artistic expression.
I don't think LoTR is totally ruined by racism or something (especially as Sauron's human allies are mostly a sideshow) but it's got some issues we'd be rather critical of if someone tried to write that kind of thing today.
A huge chunk of the audience are only aware of The Lord of the Rings
through Peter Jackson's adaptations. As far as I'm aware, nobody has complained that he didn't include token black characters (Who would likely have to be paragons of good otherwise it's racist?
Some people will always like being upset, it's not up to artists to aim to please such people.