I don't think he was miscast, I think the character was mis-written. The film lost me when Smith actually said he was evil. That's a silly way to write any villain, and it's out of character for Smith. He didn't consider himself evil. He was just greedy, self-centered, and narcissistic. His priority was to benefit himself, and if he had to deceive, manipulate, or hurt other people in order to increase his profit, comfort, safety, and personal fulfillment, then so be it. And he did have enough stirrings of conscience to impose limits on how far he'd go.
As I've said before, the closest thing we've had to a modern version of Dr. Smith was Gaius Baltar on the Battlestar Galactica reboot.