I've long thought that if they hadn't followed TSWLM up with Moonraker, just to cash in on Star Wars and had instead made FYEO (as was the original plan), that would've been the ideal time for Roger to quit. He'd have made 4 Bond films, 3 of them really good, TMWTGG being his sole dud (and it's not awful by any means). He'd also have quit before he got ludicrously old for the role.