I don't think the Doctor is prohibited from visiting them, but he is prohibited from having adventures with them and adventures have a habit of finding the Doctor. So it's more a thing of prudence to avoid them than a hard and fast rule (hence why River was able to visit them).
BTW, for all those who are complaining about the Statue of Liberty. I was too busy thinking "Oh shit, the Statue of Liberty is trying to get them too" to think of whether it made sense. Seriously, possibly the most iconic structure in America is trying to kill them (with the iconic angry angel face). That's worth including even if the only logic is "they're both statue-like things."