I really don't see how it'd be a violation. At least, not any more of one that when someone masturbates to a picture or dresses a sexdoll up as someone. I mean disturbing, yes, and certainly not to make the person being copied fell to good but, "ethically" not too much different than any other form of masturbation.
Unless someone signs a model release or places them self in the "public eye" it is illegal to publish images of them. Now imagine someone making an exact animated replica of you and having sex with it. It is more than just masturbation, it would be pretty much exactly the same as it would be with that person. So if that person never gave you permission to have access to that, I would consider it a violation.