The weight would still be there. Opening up a vertical hatch should allow for the same forces to work to eject the core. Perhaps even a hatch that opens at an angle towards the rear of the ship allowing the ships forward momentum to help get it away from a ready to detonate core. And the whole thing can be done mechanically or explosively (blowing the hatch). The angled hatch could have a spring at the forward section of the core to help push it away from the ship too.
I'm wondering if a horizontal ejection system through the rear of the ship might not be in the best interest of the ship? If a horizontal warp core ejected backwards then the core could run into the ship (if the ship slowed down for whatever reason.)
However, if it was ejected perpendicular to the ship's motion it should clear the ship in most cases. (Much like a pilot ejecting from an aircraft.)
If the core were angled and it was a diagonal core it could be interesting but also unconventional for getting people to repair or maintain it in the ship, IMO.