Just rewatched it.
I looked for why Rory went back to the gravestone.
It wasn't so arbitrary as some people here make it out to be.
He walked past the stone and he seemed irritated by something he saw out of the corner of his eye and went back there because it kept bugging him.
I can easily imagine doing the same thing if I walked past my "own" gravestone, or even just one with my exact name on it, without realizing it right away.
^Yeah I don't see why people have a problem with this. Time itself kept drawing them back to the graveyard, why is it such a stretch that time itself would draw them towards a particular gravestone?
Unfortunately, that's not how the scene comes across. Darvill's glance to the side looks more like he's looking at a bird flying past or a squirrel skittering around in the distance rather than the gravestone he's passing. His weird low IQ line delivery during the "look my name" bit doesn't help either. The scence probably would have played better if the gravestone had been in front of him the whole time with him only noticing the name on it when everyone else was getting into the TARDIS and his displacement was done offscreen.
More or less something like this;
Doctor: Let's go to a pub in the TARDIS.
[The Doctor and River go into tha TARDIS with Amy starting to following behind them as Rory notices the name on the grave stone in front of him]
Rory: Hey, this guy had the same name as me.
[Amy turns around]
Amy: Rory come on...
[Rory is now gone with an Angel pointing at where he was]
It's still a contrived way to get rid of Rory but, at least it would have played out a bit smoother than the odd moment of him wandering to look at a gravestone in the opposite direction for no reason that we got.