I blame Beltran. You can be a peaceful man, not given to conflict and yet still project great strength and resolve. This wasn't the impression given by the actor. I don't think the scripts themselves present him as the mushy personality a lot of people perceived and in the hands of a better actor he would have come across a lot better.
I don't blame him at all only for the fact that its been said that Jeri Taylor asked for the male roles on Voyager to act more passive in order to make Janeway seem stronger. So in essence, Beltran was playing the part as it was written. Mostly the only times Chakotay is aggressive are in eps. Janeway isn't in charge of the ship.("Scorpion pt2", "Timeless" or "Fury" for example.)