Everything you're saying to me simply sounds more like a problem within the fanbase than the films, people not wanting even the slightest variance from the what they expect and feel safe with. I don't want to feel I'm watching something where there's no creative freedom, where the plot is defined by a formula and anyone one of us could sketch it out in advance because that formula is so well known and goes so unchallenged.
I know I'm harping on the point but if we are applying the HJ to Luke he's already completed it, it can't define him so what is supposed to happen next? If we are to follow the conventions established in the world building that has gone before one side gains temporary ascendency before falling, typically due to some internal strife or failing amongst it's most prominent figures. That cycle works in both directions, The Empire fell due to Vader's questioning of Palpatine and guilt at his own actions, the Republic before it due to the hubris and arrogance of the Jedi Order and their failure to predict Anakin's fall, prior to that we know of Plagueis reaching previously unknown heights within the Sith teachings, much as Yoda later did with the Jedi. He was killed by his own pupil. The SW universe is on the large scale defined by the constant flux between two polar extremes and the shift from one is almost always internal. Luke as seen in TLJ is just the latest iteration of that.
The problem is people have grown attached to him above all other characters, he has come to define the idea of the ultimate hero of the setting, the final victory of light because that's how he was presented to us in the OT. Everything else has stemmed from that and comes secondary in the minds of much of the fanbase.
The world doesn't neatly wrap up adventures though, nor does SW. Life goes on and it does so without the benefit of a compass defining the route. Therein lies the problem, the story had to move on beyond the events of ROTJ and that means seeing what happens after the glorious victory, what comes next and how reality never meets the hopes and expectations of those who fight for those ideals. Luke being disillusioned and living in self imposed exile is well within the bounds of valid story telling, but that doesn't fit with the image people have of the happy ending he came to represent.
It’s not life. It’s fantasy. And Luke’s little section of it in particular is one that found its home with kids. Star Wars, essentially, is family viewing, shifts mountains of toys. Five year olds and thereabouts. It needs realism in its mainline stories about as much as the Godfather needs a few song and dance numbers a la Busby. Luke absolutely is that shining light.
Maybe it is safe, a little, but it’s certainly not a hinder on creativity for any creator worthy of the title.
The problem today is the nihilism turning up in many things. We call it post modern, we call it deconstruction, we throw all these nice criticisms at it from our media savvy adult world. But when we were kids, the heroes won, the baddies were defeated, good triumphed over evil. Where do today’s kids get that? Cos it won’t be in Star Wars where we got it. It’s not in Star Trek, cos that ain’t kid friendly anymore and it takes fifteen episodes of non kid friendly stuff to get there. Doctor Who? Maybe. Except we had a basically suicidal Doctor refusing to regenerate on Christmas Day while he visited by everyone he knew...and who were all dead.
When does the hero, the one the kids whoosh around pretending to be, get to live and be happy?
Because sod real life, I get real life in my real life, I want kids to aspire to be good, and live, and be happy...we don’t need to teach little boys and little girls that it’s all about dying for cause, before we teach them to live for a good one first.
That’s the first rule of these things: good triumphs over evil, the dragons can be fought and beaten.
It’s one we deconstruct into ‘there are no dragons, justa different point of view’ because we are such clever adults now.
That’s why Luke needed to stay a hero, though I do not think TLJ is quite the mess some people think, nor is it the glorious ‘anyone can be the hero’ others think...not least because who wants that job? You get to die alone with that job...it’s happened to every last one in the story now.
You can absolutely have a character be the obi wan, absolutely turn the wheel, but any creator worthy of the title knows how to play the game. Lucasfilm needs to fin one...even Rebels got dark, and lost the kid fans. It’s irresponsible to promise a story of hope, and give one with none.
People took Clerks little Empire discussion a little too much to heart.
Life is not a series of down endings.