The biggest barrier Japanese animation faces in the west is the culture barrier. People tend to be overly sensitive and judgmental towards different cultural mores. You don't have to go to Japan to find dramatic culture clashes with America. Go look at European popular culture and television, and you'll find in many places, that the wanton violence and male testosterone exploitation take for granted in America is frowned upon, but topics like sexuality and displays of sensuality are far more accepted than in the sexually immature, neo-puritan North America.
A problem Japanese animation faces in western translation is what Carl Macek, pioneer of localizing Japanese content in the US, called "ethnic gesture". Japanese animation is full of specific iconic cues, expressions, body language, that means a lot to Japanese audiences but can seem like random jerky nonsense to Americans. Japanese stuff tends to be super punchy, hyper active, and absurdest. Partly in response to overly dapper, restrained, polite Japanese culture.
Japanese animators have also traditionally used these tricks and techniques to get around limited budgets while still having characters with a lot of expressiveness. Americans may appreciate only what they see as technical quality - animation with a steady amount of consistent frames per second of action, never deviating. Japanese animators throw that to the wind and do whatever it takes to achieve a certain effect, and save time and budget for where it's really needed. This makes low-to-medium budget animation from the east look inconsistent to western viewers.
One thing that many don't realize in America is that much, or most, Japanese animation is adapted directly from Japanese comics and graphic novels. The printed material acts as the initial guide; relatively few series are entirely original, starting as animation first, or are created without a companion comic series. Truthfully, this has an advantage - with the entire world of Japanese comics to call upon, there's a much wider and more eclectic variety of source material to call upon.
Westerners have narrow stereotypes of what "all that anime crap" is, because of a handful of genres and even specific titles that were the most exposed during the original western anime boom of the late 1990s and early 2000s.
In western animation, there's a very narrow range of genres that ever get approved for creation. Even today, there's a big invisible wall around American animation for first-run television. If it's not a cheaply animated sitcom, comedy variety act, or a super hero action adventure either for kids, or straddling the line of kid-adult... there's no place for it. There's certainly no place for shows that practically have no genre (as often seen in Japanese animation.)
I suppose as well, another reason that Japanese stuff doesn't sit well with some Americans is that different cultures have different basic philosophical underpinnings. This affects small details that can make anime seem "wrong" - things like the way people phrase things, basic manners, even basic logic in how expressions and thoughts connect together. It all becomes odd when run through a layer of translation, like dubbing or sub titles. Most foreign media that Americans are used to watching via translation is still of western origin - be it french, italian, or even latin america. To American eyes, a dubbed french film is merely different enough to be charming. But making the jump to an asian culture is much bigger.