So, it's even more likely that things happened that impeded their ability to conduct their follow-up mission. Yeah, granted, the Ancients first arrived at earth some fifty million years ago, and the Ori only found them five to ten million years ago, leaving what is, in historical terms, a long-ass time for them to check up on the Destiny. But in terms of periods that long, it's entirely possible that the Ancient civilization had multiple rises and falls and we just know about the most recent one, which was caused by the Ori Plague. It's kind of unlikely that the Ancients enjoyed 40 million years of complete stability. (Incidentally, I once thought about it, and it literally boggled my mind to imagine being in a society with in excess of fifty million years of recorded history. God, their AP exams must be murder. I mean, I have enough trouble keeping 43 presidents straight, never mind if there were ten million of them).
Or maybe the Destiny had a crew, and they died or abandoned the ship without turning it or the pilot ship off. Fifty million years is, after all, a long time, and the Ancients didn't seem to believe in putting away their toys after they were done playing.
