It sounds to me (limited experience here, working in group home with similar disabilities for 7 years now) like you're not getting his attention, and maybe not being firm enough. With my guys I find it's hard to get them to listen. One is very skittish and will always tend to walk away from you if you try to talk to him. But even the deaf one I can get through to on most things.I have told him several times I am not interested. I have even snapped at him when he told me about it for the umpteenth time but he won't stop.
I don't agree with those who are saying ignore it. What good does that do him? What does that teach him? For some things ignoring works - behavioral things - but this IMO is just finding a way to get his attention, explaining it simply, repeating the explanation a few times, getting a response from him that he understands, and then just nipping it in the bud and being very firm and consistent.
I find these guys are (usually) plenty reasonable when they get the information. You can even explain about the alone time in the morning if you can get him listening. Then you can throw down some praise when he gets it right. Hope this helps.