Chakotay and Seven pairing up didn't bother me in the slightest.
Consenting adults have the right to choose to be with whomever they want. People often choose those who others don't think are right for them. That's real life. I see this as an instance of that. If these characters like each other that way, then that's who they are.
The surprise aspect of not really seeing it coming is also akin to real life. It's really no one's business but there's, so why should anyone else see it coming?
Playing it that way, as something between Chakotay and Seven that only they understand, neatly torpedoes everyone else's interest in who Seven ends up with and makes her her own person. Doing things for her own personal reasons is a nice completion of her arc to become human again and an individual. I thought Seven covertly beaming into Chuckles' quarters was kinda cool. Clearly, that underscored the privacy angle.
So, I guess I really am in the minority!