Well, gee, we are talking about at least a couple of different things here:
1. Do I have a right to judge his personal relationship with his wives or and lady friends? Nope. I do not. That's their personal business.
2. Do I have a right to think he's a slimeball if he forced subordinates to have sex with him - you know, the old casting-couch routine? Yep. Mind you, I'm not saying he did - I don't know. But if he did, and there was good evidence about it, you're damn right I'm going to judge him, and my judgements will not be kind. I wouldn't expect any different if it were me acting like a slimeball.
3. Do I have a right to decide that his vision of temporary marriage is not one that I hope to see for humanity? Of course I do. You have a right to judge it, too. That view, unlike his relationships with wives and lady friends, is open for public discussion. He opened it for public discussion by bringing it up publicly.