This is pretty classic harrassing behavior, which is the early stage of stalking. If you have truly been that direct with him, and he's still acting this way, you need to take every step possible to protect yourself before this escalates...and it will, quickly and dangerously.
By all means, tell his wife, but as others have said, in most cases this won't get you anywhere. If it does resolve it for you, consider yourself lucky. If it does not, swift police action will be necessary to set up clear boundaries.
Remember, to the stalker, any
attention you give him is feeding the disease. Even bad attention is attention. You don't have to live in fear, but you need to cut off all contact with this individual, ASAP. If he has your phone number or e-mail address, change them or block him. If he looks at you, do NOT make eye contact. Don't exchange even pleasant greetings with him. Don't attend or throw any social gatherings where he may attend.
I do understand that this probably doesn't seem to you to be as serious as we're all describing it... but you have to be very cautious. As I said earlier, these things escalate very quickly.