So here's my question. Obviously I want this hairy little beast to leave me alone and it's a shame too as he was a good neighbor to talk to from time to time...but anyway I was thinking that if he asks me this crap again I'm going to say "what part of we're not hanging out do you not understand? If you can't comprehend this then I'll be forced to go to Jean (his wife) and have her explain this to you"...
I think you are on the right track here. How do you read this guy, though? A low-risk pest or clueless wonder, but otherwise physically harmless to your well-being? You just need to anticipate how you think he may react if he calls your bluff or forces your hand--which would send you singing to his wife as threatened if you make good on your word.
On the one hand, if you do go to her he is liable to be in all sorts of trouble on the homefront, maybe even finding his shit out on the lawn one day and his bank accounts empty. Do you want to be the trigger for that--at least in his mind--and thus the target for any crazy schemes or "retribution" or physical escalations on his part?
You need to consider your own safety if you are commited to standing up to this douche and elevating the matter to his wife. On the
other hand, NOT following through on your threat may just further embolden him, or make him think you are playing "hard to get", or some other sad confusion on his part. You don't want some nut after you, either in unrequited lust or a fit of anger or even rage at what "you did" to him.
Personally, to a woman in your position, I would recommend caution. However, if he comes into/near your home again without your consent, you
must call the police and report him for tresspassing. This might be the ideal solution, since from his past behaviour he only gets brazen with you when wifey is not around--and
THAT is when the cops need to show up.
He'll get the message that his advances really
are unwanted, the police can get a report from you about what has been going on so that they have some history of the matter, and the wife will not be alerted (yet) to his indiscretions towards you. If you do it just right, you'll have him just where you want him--neutered and scared away from you, but not "found out" by his wife and giving him his potential big beef with you.
. A stalker will stalk no matter what...they are compelled to... and little if nothing will get in their way. I have to agree with Juan's description. 
