I have, with the help of my step father who is an expert in all things nuclear... sat down and defined what happens during a core breach.
First of all there are multiple triggering events that can lead to this situation.
First and foremost is loss of cooling to the chamber field generators. As those generators heat up field output drops and the pressure/heat of the reaction can push through in an uncontrolled manor, causing the ship to go FOOOOM.
Second would be the collapse of the articulation frame, this would result in an uncontrolled mix, a surge of power and the failure of containment.
Third, physical failure of the pressure-vessel and destruction of the field-effector elements. Think like a disruptor burn or a bomb.
Finally, exotic particles and subspace stuff. Who knows what really goes on in the pretend-core? The writers do, and if they say muons can cause an explosion so be it.
We also worked this out in regards to the Galaxy Class's inability to eject the core right:
Cryogenic slush/fluid is circulated from the main matter tank down to the reaction chamber where it is circulated in "water jackets" around the core, it picks up heat. From there it is piped up to the injector and shot down the constrictor segments and into the core to react with antimatter.
The fuel system in effect is part of the cooling system. Shutting off the fuel flow causes a loss of major portion of the cooling system. Shutting down the core to a standby/cold-iron mode requires at a minimum 45 min or more. In an emergency situation the antimatter system can be shut down and the matter system continues to run producing normal "fusion" plasma and that is gradually ramped down.
In extreme cases the system was to cut off the flow of fuel and coolant, purge the conduits and launch the core out of the ship at very very high speeds. Practically speaking the conditions that would allow that to happen never really occurred.
During the events of Generations, LaForge (being THE Galaxy-class expert) knew that there was no way to shutdown or eject the core in time. This lead to the evacuation of the ship. I can see this being SOP after the loss of the Yammato, rather than risk a core ejection attempt, get everyone off and then try... dialog suggests that the core breach happened ahead of the countdown, which is why the saucer got knocked out of orbit.
Later ship designs used a less complex cooling/fuel system and included additional layers of redundent protection so the core could be disconnected and ejected nearly "at will."