Then why was he trying to date Seven of Nine in Relativity?
Temporal Prime Directive (which doesn't exist!) to one side, the second Carey recognized Seven of Nine for who she was in "The Gift" after they took the aluminium siding off her face, no wonder he went into hiding concerned that he would cock up the timeline if Seven never went back in time to give him a boner.
Although if he recognized her immediately in Scorpion?
Would he have told Janeway about this cyberwoman's future to join their crew?
That they could flip her.