I am exploring this topic in my own fan-animation and have therefore done some research.
In the episode of TAS "The Practical Joker" the ship's computer starts acting wacky and even starts laughing over the intercom. So I think even with the older duotronics, that the computer could approach sentience, but was never allowed to really exhibit it. I think something like this happened on Voyager too, with the computer getting "sick", since it had the biological gel-packs.
The M-5 was a disaster as it cost lots of lives. Nevertheless, it didn't destroy Richard Daystrom's legacy as the "Daystrom Instute" became the skunkworks for Starfleet's computer systems going forward. Nevertheless, aside from rogue holodeck simulations like Moriarty and EMHs (which I never liked in the first place) the ship's computer never approaches sentience.
As far as the ship being able to run itself in automation, even back to the duotronics, we saw Bele take over the computer in Let That Be Your Last Battlefield, and then we saw Scotty set up the ship to run by wire in Search for Spock. (A ship run without a crew would not do well without also figuring out a way to automate maintenance, hence the Enterprise was a sitting duck after a few hits.)
My opinion is that the computing power was capable of strong A.I. and that things were definitely headed in that direction in the TOS era but after the M-5 incident it was not allowed to do so. It's one of those lines they won't cross like Asimov's three laws of robotics.
My speculation which I'm exploring in my fan-fic is that in emergency situations the computer could be given autonomy
, but doing so would require some elaborate unlocking procedure. Of course, if you were so down on your luck as to need the computer to take over, you might not have enough surviving officers to do the secret handshaking required, but that's why I am doing my stuff as a comedy. Incongruities like that just set up punch-lines.
What I think is even more curious is why androids were not used more often in Trek, or at least robotics. In Measure of a Man, it's stated that an army of Datas who had no control over their fate would be used like slaves. However, simpler devices could have been used for more routine tasks and we see none of this. We see androids in various Trek episodes but aside from Data they never become part of the crew or alien crews, either in an official capacity or part of the overall machinery. This and the absence of "Fighter" craft is part of what makes the Trek universe feel different from other science-fiction.
Ship in a Bottle asked the question of whether software (in the case of the Moriarty Program) had any sort of "rights" and one could ask the same thing for the ship's computer as a whole if it were allowed to exhibit true autonomy rather than just serving up Earl Grey tea and searching through the historical database. Trek went to great lengths to establish that the ship's computer was much more advanced than anything we have now, but also held one arm behind its back, so it always seemed to be overkill for what it was asked to do.
When you think about the conversational interface in something like SIRI that we have now on the iPhone, you'd think that the ship's computer would at least have been able and willing to engage in social chit-chat but you never see it doing this. When you talk to the ship's computer it only responds as minimalistically as possible and without any sort of sass or personality. Running the holodeck and the personality simulations within is the closest you see to this. Just as Majel Barrett got rid of the falsetto monotone between TOS and TNG, that if you re-envisioned the ship's computer these days that you would incorporate some sort of personality profile at least when interacting with the computer in your personal quarters just so it didn't come across quite so cold. Something approximating, maybe, KITT from Knight Rider. It just seems like that's where we're heading with A.I. in the real world.