I think that the all encompassing reason for all of these security lapses can be traced back to GR's original Utopian vision of the Federation. Which of course got shot to Hell later on. On Earth, two to three hundred years from now, there is no poverty, disease or crime. Evryone is happy and trustworthy. Anyone who isn't is detected early, psychologically tested and sent off to rehabilitation colonies. As a result there's no security monitoring done, everyone can go where they wish, and there are no locks on the doors. Ships only have security details onboard to deal with any potential hostile aliens that they may meet that haven't signed on to the big Federation lovefest. And what happens is children wandering onto the bridge, aliens stealing the ship's most vital dylithium crystals, and all sorts of people wandering into main engineering. Main Engineering on the Enterprise-D didn't even have a door. A main corridor just dumped right into it.