He may have. But his duty is to protect Asgard, not Midgard. That and the fact that the Elves seemed to travel by something closer to conventional space travel makes things more confusing. However, it's honestly not something worth overthinking. The Nine Realms are unique because there are nine of them. They're different from the hundreds of thousands of habitable planets in the universe. I wouldn't think about it any more than that.