I don't necessarily disagree exactly, but the postcard message from Fitz and the suspiciously specific prophecy about a group of people who were not generally known to the public seems to indicate he's the one behind a lot of the goings on we've been seeing. In order for him to do that, he'd need both knowledge of the future and the ability to act in the past.
Of course we still don't know who Mr Rubberskin is or what that group's agenda is, much less why the team was sent forward, nor why Fitz "wasn't on the list."
If I had to guess I'd say the show is sticking with the "you glimpsed the fourth dimension now you're stuck with it" idea from the season before last (remember the Inhuman would could see a person's death?) Which would mean those aliens posing as government agents are acting out something that they already know has happened and so is Fitz. The question is, can they break out of a causality loop?