Stephen Hawking did famously say "If aliens were to come to Earth, it'd be like Europeans landing in America, and that didn't work out very well for the Native Americans did it?"
Not necessarily. Maybe it takes too much energy to get here for it to be worth coming here for the sake of conquering. Why come halfway across the galaxy to blow us up? You gain nothing, you would have to colonize and control us. And even with superior technology, how do you colonize and control seven billion people? You'd have to bring an army, and that'd have even higher energy demands.
Also, these aliens would have had to pass pretty damn close to us, in the last few decades, to even know we'd developed.
I don't believe we're the only intelligent life in the universe, but it's probably sparse enough that it's incredibly hard to find us and incredibly expensive to reach us, even if warp drive exists.