It was easier back then (Anyone could write a script on spec.)
Now, the WGA rules are much stricter. A nonunion writer can't do that anymore.
Of course they can -- by joining the union. That's how it works. A union that refused to let in new members would be pretty useless.
There are actually a fair number of prose SF authors working in television today -- often as executive producers and writers when their books are adapted to the screen, like Daniel Abrahams & Ty Francks (aka "James S.A. Corey") on The Expanse or Martha Wells on Murderbot. George R.R. Martin of Game of Thrones has been writing for television since the 1980s, alongside his prose career. Novelist John Scalzi was a creative consultant on Stargate Universe. But the only one who's been involved with Star Trek was Michael Chabon, who co-created Picard, and he was let go after one season.
So it's not that it doesn't happen. It happens plenty. It just doesn't happen much on Star Trek anymore. And there's no reason why it couldn't.