Aw come on, he smacked Jan ONCE while having a psychotic breakdown (and this was right before he defeated the Masters of Evil singlehanded) and it's his mark forever. No one cares that Jan used his mental breakdown to get him to marry her. They're equally bad if you want to have it that way.
And, I might add, she divorced him immediately after. No halting long period of excuses and relapses--he was gone. Until someone retcons it as 'not being the first time', a hero had his lowest moment, and paid his highest price.