Saw a (cartoon) movie where Snow White, Cinderella and Sleeping Beauty while talking up their fiances in a bridal store, all realise that they are engaged to the same bloke.
Something similar could easily happen in the Bond universe.
He's a freaking secret Agent!
Jim could easily have five families scattered across the globe unaware of each other, and the wives and children are blaming his very-very important job for why they hardly ever see him,