Actually, the conference room (or Observation Lounge, as it also was somewhat confusingly and apparently accidentally called) could be accessed from many different directions. At the starboard end, there was a door that supposedly led to the door we saw at the starboard aft corner of the bridge - but not directly, as there was a piece of corridor behind this door on the conference room set, and one early episode even showed a steep ramp along which Riker and the guests descended to this corridor and then to the conference room. This corridor continued aft (to the right of the door as seen from inside the conference room), supposedly leading somewhere; the Sternbach blueprints indicate there's a further ramp there, going down to Deck 2 without visiting any compromising locations.
On the port end of the conference room, the door similarly leads to a corridor with both left and right options available. Right/forward ought to terminate into the turboshaft that feeds the bridge port aft corner turbolift; another risk-free way of delivering people to the conference room. Left/aft is again indicated to be a ramp down to Deck 2, reinforced by the scene (I forget the episode) where a couple of kids are seen playing in the conference room, and our heroes shoo them away - the kids dart to the left, indicating there's indeed an escape route there. Darting to the right would make no sense as the kids wouldn't stop to wait for a turbolift...
So the conference room is risk-free VIP territory. It's just that Picard often allows his VIP guests on the bridge, supposedly as a gesture of trust. If he didn't trust his guests, he'd no doubt take them directly from the transporter room to the nearest turbolift stop, then take that lift directly to the portside door of the conference room!
Letting the VIPs into the Ready Room is the ultimate gesture of trust. If Picard wanted to keep ship's security intact, he'd simply receive his guests elsewhere. Indeed, it would be a bit silly and even insulting to take the guests to what is essentially the closet where Picard catches a quick nap, opens his bottle, or rubs his crotch, during those situations where he can't leave the bridge for more comfortable and representative locations. But the insult becomes a compliment when Picard takes his VIPs to this facility despite this compromising not only ship security but also Picard's privacy!
Who did Picard receive in his ready room that he shouldn't have, really? He had Gul Macet there at one point, but he also made the important propaganda point of allowing Macet access to his bridge during combat operations - and indeed letting Macet use the ship for hunting down another Starfleet ship! Picard's other guests there seemed to be quite benign, mostly being trusted Starfleet/UFP officials or allies.
Timo Saloniemi