...the kid who talked to the Doctor replying, "Are you thick or something?". Not sure, but I don't think "thick" implied "dumb" in 1851.
The TARDIS translated the idiom for the audience.
Furthermore it doesn't make any sense for someone in 1851 to know minutiae about the Doctor, like allons-y. Not unless he's from a more advanced civilization who's had contact with the Doctor.
There's always Torchwood. Granted, the Doctor hadn't started saying "allons-y" at his first encounter with proto-Torchwood, but they could've gotten information about the Doctor's other pre-1851, post-Doomsday adventures.