The filmmakers treating Khan like he was some great Joker-tier antagonist really did feel out of place. The closest Trek ever had a recurring nemesis was with Gul Dukat, and that was for DS9. It sort of the same thing with Darth Vader starting off as a really cool henchman with only one movie where he was really in charge (Empire), but for after awhile the perception of him being the biggest baddest thing in the galaxy exploded, with the prequels going by that sentiment rather than having Anakin's story be something of a tragic subplot.
The public at large wouldn't know Gul Dukat from a seagull.
Unlike Vader, Khan was never a henchman. He was the bad guy in two previous Trek installments.
Harry Mudd appeared in three Treks. Q in several episodes over three different series.