One argument in favor of many if not most of the Excalibur crew surviving is that we don't know of any mechanism by which phasers could kill the crew of a starship while leaving the ship visibly intact.
Depends on "visibly intact" but we we do have two examples.
"The Wrath of Khan". Phasers at low power and close-range left scorch marks (at the weakest) and shallow cuts which would be difficult to make out at the size Excalibur was displayed at in the original FX. In TWOK, these shallow/light phaser hits with shields down resulted in alot of internal damage with casualties. It is very possible that full phasers at medium to long range can cause massive internal damage without significant visible damage. Full phasers at close range would result in instant destruction, as seen in "Day of the Dove".
And IIRC, in "Basics" the backup phasers were overloaded and discharged inside Voyager, stunning the Kazon crew in one go from the inside of the ship and through the bulkheads.
As to whether there could be some survivors on the Excalibur, it would depend on how much time Wesley's surviving ships had to beam off the survivors before they retreated out of Enterprise's phaser range. It doesn't seem like any shuttles were launched or people left via some sort of escape capsules as that would have likely been mentioned. And as pointed out, M5 is smart enough to detect the level of destruction it has wreaked.
Regarding difficulty scanning for life forms in TOS:
"The Doomsday Machine" - we are told of heavy subspace interference.
"For the World Is Hollow and I Have Touched the Sky" - it is possible that the Fabrini outer shell prevented accurate detection of life or Fabrini don't register as normal life signs.
However, scanning another ship like the Constitution shouldn't be too hard and there were no external sources of interference. M5 should've gotten an easy look at Excalibur, IMHO.