Why is it that alien life has to be more advanced than us? If there are alien civilisations out there they could be centuries or millenia behind us in development, making us the most advanced species (scary thought when you take time to consider it).
Even if there are species out there who have developed warp drive, it isn't a means of instantaneous travel and still takes time to get from A to B. They could be far enough away from Earth that getting here to colonise our world would be impractical. That is even accepting that they breathe an oxygen-nitrogen mix, require Earth gravity (their homeworld could have a third of ours or six times as much), or can survive within our climate.
As for building an empire, why? Not every species would have our bloody (as in actual blood) history. Seizing new territory and conquest may not hold any interest for them. They may have gone to the stars millions of years ago, seen no other intelligent life around and returned hom to focus on their own development and evolution. Or maybe they're sitting up in orbit watching us after seeding our world billions of years ago, there is after all no way of knowing that didn't happen.
I like to believe that we are not alone in this universe, that other life does exist and that it isn't merely out to screw us. Look at how many types of life forms evolved under a huge array of conditions and locations across our own little planet, then just think of all the others out there. What life there is here can't be all there is.
Plus they could use some kind of starburst, or hyperspace, or probability drive to travel through space, it doesn't have to be warp