Did you read my second sentence? If he had come back for Avengers, would you still make the same argument?
No, but that was before the Power Stone was shown to fry people when touched without being in some kind of thing.
Plus since he didn't show up he means he was vaporized or possibly sent somewhere else away from Thanos aka stranded.
Nick Fury was able to handle the Tesseract in The Avengers without being teleported or killed.
And the Aether was absorbed by Natalie Portman without any issues.
The Tesseract =/= the stone. The Tesseract is like the Orb or Loki's staff - a container which anyone can touch and, potentially, activate intentionally or unintentionally. Once the stone is removed from the tesseract (which we haven't seen yet), then it will probably be impossible for any mere mortal to touch it. The Aether would seem to be a unique situation in that it is a 'container' that seems to be in the form of pure energy rather than matter. Of course we still have no idea what the last two stones will be contained in.
Actually we don't know that about the Tesseract, the Red Skull wasn't able to touch it, yet Fury was able to handle it. And if you notice Thor and Loki didn't touch it to teleport back to Asgard at the end of the Avegners movie.