The issue there is that in all of the proceeding scenes of the Doctor traveling backward through his own timeline, we're shown that he couldn't physically be observed by anyone in the those events, due to him traveling in the opposite direction in time to everyone else. It appeared that he also couldn't physically interact with anyone either, which makes it rather odd that he can even pick young Amelia up at the end there.
He physically interacted with Amy twice in the rewind. First, during "The Time of Angels" -- he holds her hands, talks to her, and kisses her on the forehead. Then, when he reaches events from "The Eleventh Hour," he picks her up and puts her in bed. Arguably there's a third time -- he shouts at Amy after events in "The Lodger" and she hears him but can't see him.
If you want an explanation, I'd venture that the closer the Doctor approached "The Eleventh Hour" in his personal timeline, the more physically present he was with the rebooted universe.