I don't think there's any direct contradiction between how Simon said he got River out in the series and what we saw in the movie. He simply omitted the fact that he was the one that went in after her while those "others" he paid to help him are the ones that got him out. In fact we see in the movie a ship come and hoist them out of that shaft and there's no way he could have just walked in there on his own, it's clear that he had a considerable amount of help.
As for Mal's character, he's clearly slid back into a similar rut he was in during the pilot, thanks mostly the the departure of Inara and the increasing pressure of keeping his ship in the air. As the movie goes on things just get worse (specifically the wholesale slaughter of just about every friend and ally he had left), so he's hardly going to cheer up along the way. Nevertheless there are a few moments of levity, so it's still the same Mal.
I agree. I didn't notice any changes in characterization that weren't warranted by the circumstances. The relations between Mal and the Tams were always tenuous at best and there has always been a fight inside Mal between his pragmatic and altruistic sides about them. Work being pretty sparse between the last episode of Firefly and Serenity hasn't helped matters. Also, Wheedon and his writing staff are human and continuity glitches are expected. Sometimes they forget little details and sometimes they rework things after thinking things through.