I agree with you, I remember stuff from when I was 2. I just wonder if it's because I haven't been much of a drinker, they always joke about losing brain cells as if it isn't true, but then some people really can't remember kindergarten, so....
I think Picard didn't want to assume boy genius remembered it all, even though I'd expect 24th century people to be more advanced and more capable of remembering things, they teach them Calculus in first grade, don't they?
When I was a child my mother asked me if I remembered things which happened around my second birthday and I didn't remember them. My earliest memories must be from about age three I guess.