So many other people are basically banished from Hollywood at even the slightest accusation, but especially with Allen, nobody seems to have a problem being involved with his movies.
I think you're falling prey to a false narrative by saying people are banished at the slightest accusation. Usually it only happens when the accusations are substantiated by an investigation. There have been cases where an accusation was found lacking in credibility and disregarded, as with George Takei a few years ago. And there are cases that remain ambiguous and come down to he said/she said, as with Woody Allen or Johnny Depp. There are still those who insist the accusations against Allen are false.
And people should be presumed innocent. Accusations should be taken seriously, of course -- the accuser also deserves to be presumed innocent of lying -- but the accused deserves the benefit of the doubt until the question is settled. You don't have to pick one side or the other. It's okay to reserve judgment and let the truth sort itself out.