Yeah, stalking is not a matter of frequency. If someone is not respecting your privacy, that is a problem. If you do not want contact with them and they don't get the hint, that's a bigger problem.
Don't let this problem become any bigger than it is. If his wife can't keep him on a leash, call the cops. Don't even hesitate on that one.