But see, even calling that "incompetence" is still making a lot of assumptions about what sort of technology they OUGHT to have for security purposes. The limitations to what their technology can do are almost certainly practical in nature:Hmm OK I think maybe you are trying too hard to talk in absolutes. While I agree that many stories would not work if the Enterprise or enemy ships had half decent security I don't think that's an excuse to say security HAS to be incompetent and that the characters should assume that it will be (didn't space hippies take over the Enterprise ffs? That's just embarrassing)
Internal sensors cannot actually differentiate one life form from another or, for that matter, one PERSON from another;
Only doors to secure locations have access controls, and this would probably include officers' private quarters, engine room, torpedo bay and impulse deck (not much else). It doesn't seem to include the bridge for some reason, but that's probably because anyone badass enough to to threaten the bridge crew isn't going to be stopped by a password lock;
Normally, anything close enough to beam someone aboard your ship will show up on your sensors before they have a chance to do that. If you can't see it on your sensors, you probably shouldn't worry about it (unless, of course, there's a 24th century Starfleet gunship cloaked off your bow and using a known exploit to beam several officers in vintage uniforms onto your ship for reasons unknown, hint hint).
Computer systems have access controls that depend on mechanical tokens and safeguards in order to prevent an outside source from remotely compromising those controls with electronic warfare attacks (this is stupendously common in Star Trek) and therefore a person physically sitting at a console would find it surprisingly easy to override that security if they knew where to look.*
It just seems to me that you're assuming a lot of capability that SHOULD be there but that you haven't given any real thought to how that capability would really work or how it would help/hinder normal operations in a way that would make those security measures way more trouble than they're worth.
See above: if you're close enough to an enemy ship for him to beam something nasty onto your bridge, you probably already have your shields up anyway. I would even go so far as to suggest that blocking transporter signals is something deflector shields were modified to be able to do for precisely this reason; that earlier generations of them probably didn't have this capability and that more advanced transporter systems are capable of penetrating at least some modern deflector arrays (as Chief O'Brien demonstrated perfectly when he boarded the Phoenix)Since we are talking about transporter use as weapons we should look at the scientific leaps of WWII where each side was racing to improve both offence and defence to outwit the other. Given the catastrophic potential of transporters it is ludicrous so suggest that in 200 years, vessels can't detect, flag, or even pre-empt unwanted incoming signals?
And even in this case, it's a tradeoff. The modified deflector shields come at the cost of a starship not being able to beam down or up without lowering its shields. Clearly that inconvenience is far outweighed by the added security of preventing any boarding parties from coming aboard your ship in the middle of a battle.
But never from a random location inside the ship to a random location outside the ship; in order to beam something off the ship it becomes necessary to put that thing on the actual transporter pad.Scotty utilized site-to-site transport using Klingon transporters in STIV and a gangster was sent site to site in a Piece of the Action.
You're also suggesting that transporter operation is somehow trivial enough that you can zap someone away at the touch of a button. Transporter chiefs are highly trained and specialized in their task for a reason. Using a transporter to eject a hostile intruder would be a bit like the crew of a NOAA vessel using a robotic arm lift to knock a hijacker off the deck. Yes, it's something that's perfectly dooable in a pinch, but it's probably not something any sane person would add to the ship's security policy.
Again, probably because somebody noticed the tall leggy plasma-energy creature strutting around on C-deck and hit the alarm just as the probe turned and ducked into Ilia's quarters.I believe that the Enterprise detected the Ilia probe's incursion in TMP. That might have been partly because she was smoking hot but if so, one has to wonder how the ship flagged it as an intruder and not just a temperature anomaly, the suggestion being that Chekov had cameras installed in all the shower units...
The Enterprise is not specifically designed to notice when intruders are smoking hot; that's Kirk's job.

I keep wondering what "real world science" you assume they are supposed to have but don't. There isn't a naval vessel in existence that would AUTOMATICALLY detect the presence of an unauthorized person on the premises just by virtue of their being there.On the one hand, I applaud their desire to limit technobabble but on the other I lament the way they mashed real world science to fit their plot and the way stupid ideas appear smart because everybody else is more stupid.
Biometric identification and token-based security measures have their own limitations, but employing them over EVERY SQUARE INCH OF THE SHIP would be both a prohibitively expensive in terms of execution cost and computing power and would, in the end, fail to solve the actual problem of hostile boarding parties since anyone who tries to board a starship already knows that they cannot TAKE OVER the ship without the crew's cooperation (coerced or otherwise). Command codes and computer lockdown and such prevent that easily enough. The goal of an intruder is to trick the crew into logging you in to the computer and allowing you access to sensitive files (see also "Tom Riker") which is more about social engineering and bullshit artistry than technology.
The weak link of any security system is the people running it, and nobody has yet invented a technology that will prevent a security guard from being taken in by a bullshit artist.
* For your information, this is basically true of every security booth or checkpoint at every public transit station you have ever been to; every computer, every vending machine, every door in the station can be unlocked using a set of keys in a little drawer under the security guard's desk. Drawbridges, tollbooths and yes, TSA security stations at airports do the same thing. This practice is not particularly problematic since 99% of potential security risks have no idea that the keys to circumventing security of the entire site are never more than five feet away from them.