Who knows how long he's had the memory worm? For all we know, it could've been something he picked up right after losing the Ponds. In a fit of depression, he could've just given it a kiss and forgotten the whole thing.
Or, it's just been under the TARDIS console for centuries, stored with all the other stuff filed under "M". Mawdryn, Metebilis crystal, Mel...
You're under the console?