For my part, I think it's a factor that could vary greatly. If you're dealing with a culture (or perhaps several cultures) that don't like outside contact, or with whom contact might be limited due to things like the Prime Directive, then a shuttle or small craft might be more ideal than a capital ship. As a non-ship example, DS9 seemed to do pretty well with a small contingent of runabouts even though it seldom had any capital ship support (the Defiant was an exception of course).
Runabouts aren't really shuttles, though. To go back to my AWACS analogy, they're E-3s to the shuttles E-2s, or a camper van compared to a minivan. They're larger, they have small bunk areas for crew privacy, if memory serves they have small galleys - they're not as long term habitable as a true ship, but they're a lot more livable than a utility shuttle. A runabout's entire job is to be a small interstellar vessel, rather than a backup transporter. So it's not an apples-to-apples situation.
Now, okay, I can see situations where making the final approach in a shuttle is called for for diplomatic or cultural reasons. But that's hardly a day to day situation. Your shuttle is gone for a day, maybe two tops - remember, the actual flight time is only going to be 3 or 4 hours because no-one is going to launch a shuttle so far away from the system that it couldn't turn around if it had to, and your crew endurance limit is 8 hours, so you're not talking about sending the shuttle off for a week of travel. If the mission is longer than two days, the shuttle's probably not hanging around on the world.
Your cruiser is probably just hanging out in quick response range anyway, not doing anything else of note, so she probably won't need
any shuttles while your diplomats are engaged; and even if she does, she's surely not going to need more than two, leaving one in reserve. The scenario is plausible and I might well steal it for fan fiction, but it doesn't speak to a need for building larger shuttle capacity into a ship.
The same logic applies to the unusual spacial phenomenon: it might well happen from time to time, but it's not a daily occurrence, and while you have crew stuck out there in a relatively slow, fragile, unconfortable box - which, again, is not going to be for more than
8 hours - the cruiser or surveyor that launched it is going to be standing by as close as possible just in case. They aren't going to be farting off to scan the gas giant on the other side of the system, so it doesn't matter that 25% of your small craft are employe don this mission. You need
some shuttles aboard, in case the situation arises. You don't need an entire Cold War carrier air wing, because you almost never need more than one or two at a time.
Heck, even the Imperial Star Destroyer you mentioned doesn't carry more than a half-dozen utility craft, according to most sources. 72 TIE Fighters, sure (we're back to a Cold War aircraft carrier with that number, of course, and that was invented by WEG - I don't know that we've ever seen an ISD launch more than six TIEs on screen) but usually only 4-6 TIE shuttles or assault landers, and 0-2 Lambda transports.
The sensors that you can mount on probes/satellites will pail in comparison to a Shuttlecraft mounted one, which in turn will pail to the one mounted on a Starship, which in turn will pail to a Starbase or land facility.
Will they, though? The sensors on a modern recon drone don't pale (as in colourless or insubstantial, not pail as in bucket) compared to a Blackhawk or Griffon. The only real advantage the helos have over drones, in terms of information gathering ability, is that the crew is on-site and not watching through a remote camera.
In Star Trek, even the starship's sensors are feeding to displays, there's no "I'm here in person" advantage. Couple that with FTL coms and literally hundreds of people so you can have a dedicated driver for the probe, is the shuttle really going to have any meaningful advantage over the probe? I remain unconvinced.