How did the Tesseract end up on Earth anyway? In CA Red Skull says something like it was the "crown jewel of Odin's treasure room" and at the end of The Avengers it seems heavily implied its power is beyond what Humans should have access too and Thor takes it back to Asgard.
The building looked ancient enough to have been from the time of the Asgardian wars with the Frost Giants.
The war may have come to Earth, but the FG's just seemed (like Loki later) to want to rule us, having a slave army, rather than anything else, considering Humanity to be far too primitive to care about.
Thor even states in the Avengers Earth is still regarded like that today, Odin presumably used that fact to his advantage and kept the Tesseract safe by putting it into the care of a Norwegian sect that had dealings with the Asgard over the years, a secret society he trusted with things like this.
Knowing that it would be centuries before Earth even overcame enough of it's superstitious fear to even think of removing the cube from their relic-like care for it, and even longer before they had technology to do that "nudge it and make it spit energy" thing it seems to do.
Unfortunately the Nazi occult divisions during the war got there first. And even then all Hydra actually did was develop pritimitive but robust superconductors to take that energy to specialised crystals that could contain and regulate the release of that energy. All of which where shattered or destroyed during the war and by Red Skull's paranoid self destruct switches everywhere.
And if it weren't for the "Hey I'm here" shit the thing did at the end of Captain America it's possible the lower levels of contained energy release might have gone unnoticed by the realms, it being 50+ years for them to bother meaning they seemed to care little anyway.
It was better off hidden on a realm no one would care about, meaning here. Red Skull just put a bit of a dent in those plans.