Yeah, it'd be one thing if they were full of crazy Angel temporal energy that prevented them from time-traveling ever again, therefore getting stuck in the past for the rest of their lives. But the idea that the Doctor can't ever SEE them again just doesn't make a whole lot of sense.