I recall reading somewhere, maybe in TMOST, that casting director Joe D'Agosta found himself being credited in the industry for very inventive casting -- when the truth was that he was just winging it, and didn't know better! William Campbell himself thought it was marvelous that a guy from the Bronx had a chance to play Trelane, a fop.
D'Agosta often did cast performers against type, for example eternal one-note bad guy Morgan Woodward as Simon Van Gelder, at the very least a complex portrayal. More casting trivia: T'Pau was played by Peter Lorre's ex-wife! My favorite unusual casting was Paul Fix, a verteran of countless Westerns, as Kirk's first chief medical officer. He had a very long career, later turning up on "The Rockford Files."
Fix - and John Hoyt - were both network choices to play the ship's doctor. Roddenberry wanted Kelley both times, but the network was too conscious of Kelley's history of playing baddies. By the time NBC picked up the show as a series, De had done the Police Story pilot, and had tested very well with the preview audiences, so when it was time to put the show in production, Roddenberry could sell them on him being the doctor - thank God! Grace Lee Whitney had been in that pilot as well, so it was easy to get her for Rand.
One of the interesting casting stories in TATV 1, is how there almost was no James Doohan/Scotty in the series! GR was looking at a pretty bloated cast budget, was looking to eliminate a character! However, Doohan had signed a pretty good deal for WNMHGB, and had a guaranteed option for 5 or 6 of 13, in case of series pickup. So his agent 'reminded' GR and Desilu of the option - and we got Montgomery Scott!