Hell, if the guest actors were all white, you can bet we'd be hearing complaints about how the episode was racist BECAUSE of the lack of blacks.
That's not really the point, though. It's not the inclusion or exclusion of black actors which strikes people as racist, it's black actors doing stereotypically black things. And whether you and I like it or not, a black man lusting after and kidnapping a white woman, a widespread and constant fear among whites, especially southern whites, for many, many years (and even among some people today), does indeed have racist connotations.
Of course, racism is subjective, and I'm sure there are black people who never found "CoH" racist, or simply weren't bothered by the stereotypes... but the defense that "it's not racist if you make the actors white" is pure nonsense. That's tantamount to saying, "it can't be racist, because if I made it not racist it wouldn't be racist anymore." Well, duh.
Of course if you used a white actor doing a stereotypically black it wouldn't appear racist anymore. If "Birth of a Nation" had featured scenes in which whites ate fried chicken and watermelon, people probably wouldn't have cried racism. But the point is they
were black actors, they
were acting out blatant stereotypes, and it
was racist.