, married man here - for almost 25 years. That guy is fixated on you and he's fantasizing about you. You're going to have to be as blunt as you possibly can be with him. You're going to have to tell him that the next time he tries this shit, you're telling his wife. Period. No "friendly chats", no hanging out, no creepy staring, no phone calls - NOTHING.
You can't "hint" at anything - you're going to have to tell him straight out.
If ANYthing happens after that, call his wife and then go to the cops. Or a big, big friend with a bat.