Because he's not an aggressive and assertive individual. He was a "terrorist" because he was forced to be, and his resistance activities only lasted for a year or two. Compare that to Kira, who was a resistance fighter all her life and didn't know anything else.
He overall seems way too laid back to be a real terrorist or revolutionary. Everything about him early on painted him out to be "the reluctant warrior". If the Federation didn't give his tribe's colony to the Cardassians who were trying to conquistador them off their lands again
from their POV (the "kinda-white man" or "He who has big neck"). He became a rebel purely for defensive reasons.
I think he just strongly lacked fire. Remember he said his father was strongly disappointed in him over his lack of enthusiasm with learning his heritage, the ways of his ancestors. Cardassians killed his father, which would help to light a fire in him. That's when he got that tattoo and took up his people's rituals and beliefs. Once he was out of the Alpha Quadrant and realizing the distance, I think he realized his fight is 70,000ly away so why wage a battle on the other side of the galaxy when it does neither side any good. That and Seska already has his balls (how else did she get his sperm to conceive that pawn-child?). He was sleeping with the enemy and didn't even know it. The Cardassians were running circles around him. One wonders if Seska lured the Val Jean into that trap in "Caretaker". Chakotay readily took to Seska, to Janeway, and even to Annorax.
Looking around, it looks like he was in Starfleet for 20 years before resigning, so he was a career Starfleet officer. No wonder he took so readily to Voyager whereas some Maquis were very resistant. He was already domesticated long ago.