Yes, let's remember the lifespan of this death penalty rule. Captain Pike proposed it in 2254 or so, and it existed until 2268 or so when it was first challenged and found both inefficient and unnecessary. For that time, it was Starfleet's General Order 7 (or more accurately, GO7 included the ban and penalty on Talos visits, perhaps among other things). Yet a few years later, in "Turnabout Intruder", the only death penalty in the books was General Order 4 (or something mentioned in GO4) instead.
It only seems natural to me that Starfleet would encounter truly mortal threats to the Federation, its citizens and its way of life in the course of its deep space explorations. There would be an initial reaction to such threat, naturally maximized in potency to match the graveness of the threat - and then a gradual increasing understanding of the threat, which would also lead to gradual easing of the countermeasures. New threats would constantly keep emerging, though, so it might be standard for Starfleet to constantly have one or two death penalties in the books, but never for more than a few decades.
Certainly the Talosian threat was a grave one: had these little telepaths been intent on galactic domination, they would easily have achieved that goal, as long as somebody provided them with an interstellar spacecraft. Death penalty, or threat of some other sort of extreme punishment such as torture of loved ones, would sound like a good deterrent, since alternatives would be lacking. Starfleet couldn't blockade the planet because the Talosians would capture the blockading ships and take over the galaxy. Starfleet couldn't issue comprehensive warnings to the general public, either, because many would see the Talosians as a useful tool in their own plans of galactic domination or other evildoing.
The risk of complacency at the acquisition of Talosian self-satisfaction techniques was never a serious one, and probably played no role in Starfleet's decision to isolate the planet. And the risk of galactic conquest was exposed as minimal in "The Menagerie", where it was shown both that all of Starfleet's efforts at stopping people from visiting Talos were futile, and that the failure of those efforts did not in fact lead to a Talosian takeover of the universe.
Generally speaking, the Star Trek galaxy could be expected to present a great many threats that would lead to the quick extinction of mankind unless mankind responded with draconian measures. That the TV shows and movies show relatively few such threats is an argument against those draconian measures - but it's not an argument the heroes themselves could use. They can't know that the galaxy is in fact an unrealistically safe place where nothing only rated for adult audiences can ever happen.
Timo Saloniemi