Han's motivation is money right up to the end. That's why he agrees to take Luke and Ben ("This could really save my neck.") That's why he's interested in saving Leia ("more wealth than you can imagine"/"I don't know. I can imagine quite a bit" and "I'm not in this for your revolution. I expect to be well paid"). That's the main reason why he wants to leave right away ("You got your reward and you're just leaving?"/"That's right. I got some old debts I've got to pay off with this stuff"). His line about attacking the Death Star looking like suicide isn't his main reason for not joining the attack - notice how that part of the line starts with the word "besides"?
As for Jabba looking for Han, seems more likely Han was avoiding him since he didnt have the money, which is exactly what we're shown in the movie. He's not trying to 'duck out of paying'. He says multiple times throughout the movie that he needs to pay Jabba back but simply doesn't have the cash.
He does change his mind at the end and come back for Luke, but I don't think that completely explains why he wouldn't attempt to pay Jabba during the three years between ANH and ESB, especially since he's already seen how dangerous it can be to avoid that payment. He was literally warned that Jabba would put a huge price on his head. His final decision was less a choice between joining the Rebels and paying off debts as between putting himself first or helping others and joining the cause.
You said his "entire motivation" was money in order to pay off Jabba. Obviously he was worried about money. Obviously he was worried about Jabba. But he said he had old debts to pay off, in the plural; Jabba was after him for dropping one shipment, which is only one debt. So either Han was lying, being sketchy, or Han owed multiple creditors and had for a long time. Sure Han was interested in money; what kind of mercenary wouldn't be? But there's evidence, which I've cited above, that he wasn't necessarily going to pay off Jabba straight away even if he had the money. If paying off Jabba was his "entire motivation," he wouldn't have stuck around for the
Death Star attack briefing. He was going to split after the briefing, because it was a suicide mission, and he wouldn't get to enjoy the fortune he'd finally gotten.
[
edited to add: If Han's entire motivation had been only to pay off Jabba, as Han and Chewie are loading up the reward would Chewie have made a growl indicating the question "Are you sure we're doing the right thing?", or would Chewie have growled "Fuck, yeah, let's blow this joint!"? Chewie was being the voice of Han's conscience, implying that he knew that Han was conflicted about what to do.]
By the way, if Han had gone to Jabba's palace and paid Jabba, do you really think that Jabba would let him walk out of the palace a free agent? Jabba could easily offer Han more work, which he'd expect him to accept, knowing he's a broke mercenary. If Han felt he was well off enough to refuse, Jabba might wonder why. Besides, there's no reason to think that Jabba hadn't heard and/or figured out that the easy little charter Han had was to ferry the droids that the Empire had turned Mos Eisley upside down looking for, especially given the way he departed Tatooine, with three Star Destroyers on his ass. Han would know that paying off Jabba could be as good as walking into a trap to be captured and sold to the Empire.
With a fortune, he could go anywhere in the galaxy. He could set up his own operation, the way Lando had done. Maybe he does business with Jabba, to pay off his debt in a structured way, and ditto any of his other creditors. Maybe he tries to keeping staying a step ahead of all of them, the way we've been given reason to believe he might have been doing for a while.
There's a very wide space between Han's being a mercenary and interested in money and on the other hand what you said, his "entire motivation" being to pay off Jabba. There's a great deal of evidence for the former but none for, and lots against, the latter.