I agree. I didn't notice any changes in characterization that weren't warranted by the circumstances. The relations between Mal and the Tams were always tenuous at best and there has always been a fight inside Mal between his pragmatic and altruistic sides about them. Work being pretty sparse between the last episode of Firefly and Serenity hasn't helped matters. Also, Wheedon and his writing staff are human and continuity glitches are expected. Sometimes they forget little details and sometimes they rework things after thinking things through.