See what I really mean by this without being flippant is it seems like this is a component of Chakotay's personality. It is one thing to to say he's being dense or stupid in having Seska as a lover when she's clearly not a nice person but I think that easily done disservice may not be the truth. Really I think he has a submissive personality and gets some kind of emotional payoff with being bossed, ordered and commanded. Everything coming out of Seska's mouth was high handed and putting him in his (in her eyes) place. Everything coming out of Janeway's mouth is ultimately the words of the person in charge of him even if they are having a quiet dinner together. Everything coming out of 7's mouth is, well.. 7. Superior Borg speaking to less perfect man. He is looking for this kind of relationship and he's willing to put up with a not nice person because the payoff whether emotional or sexual is so good. Once Seska's out of the way he transfers his affections to the next woman who will ride his ass (and a thrilling ride it is..)
This is a component of why he handed the Maquis reins over so quickly to Janeway, she was commanding and by giving over his leadership (which was no doubt back seat driven by Seska) he was in his comfort zone. I seriously doubt he would have caved like that if Voyager's captain had been a man.