He didn't remember the overgrown Boy Scout per se, all he knew was that James Kirk was someone that is Mother dated when she was younger. David didn't even realize Kirk was his father until the events of Star Trek II: the Wrath of Khan. Carol never told David who his father was because she was afraid if she did David would go looking for him.Since David remembers the 'overgrown Boy Scout', I figured that at first Kirk would just show up randomly as duty permitted and Carol finally had enough of that part-time parent crap.