To me, if the makers of the episode viewed the pod jettison as a standard part of the operation, then they would've written in at least a little something about Finney's death being unnecessary under any circumstances, and had Kirk question himself as to whether he could've done more to make sure Finney was indeed out beforehand, rather than focusing solely on when in the sequence of events did the death occur.
And, of course, they would have called the button the "LAUNCH PROBE" button instead of the " JETTISON POD" button. I think it's pretty clear that having the pod leave the ship is never part of the routine operation of the device. It can be "jettisoned," if necessary but it is never ultimatley "launched." It remains attached, manned for ion storm readings, then the crewmember vacates when the measurements are done and the pod is buttoned up until it is needed again and it never leaves the ship.
Also, for what it's worth, a Phase One search was implemented to find Ben Finney:
"It's a painstaking, thorough attempt in and around
the ship to find a man who's unable to respond."