Actually, that rock is quite clearly the one in Central Park. In fact, all of the stuff in the park is quite clearly Central Park. The fountain, the area where Rory is taken, the bridge, all of it. It's Central Park.
And could have been shot in Cardiff with the background CGI'd in (like they did in plenty of other shots in the episode.) They only reason they went is Moffat's obsession with appealing to the tiny American audience.
No. Not the fountain. Not the bridge. It's not just the backgrounds, those are REAL places that people in New York go to. I'VE been to them. I've walked where Rory was taken. That fountain, it's already famous because of the play Angels in America.