Grace Lee Whitley? I hate to say it, but she would've sunk. She just didn't have the experience or the acting chops to match Shatner, Nimoy, and Kelley. And when you couple that with problems she was having off-screen at that time... Well, I just don't see it happening.
Someone like Lee Meriwether, Sally Kellerman, Julie Newmar, Diana Rigg, or Honor Blackman would've projected the right sort of authority for a female first officer, but I frankly doubt that TOS' writing would've been up to the task, either. So I think it's for the best that we got what we got.
Agreed. I wouldn't say this if she were alive and reading us, but Grace was never an accomplished actress. And it seems to me she got worse as time went on, with her final performance on ST: VOY being a complete embarrassment. I'd blame off-camera life stressors and a lack of commitment to the craft. Stardom just wasn't in the cards for her.
Re: your other picks, I would endorse Lee Meriwether and Sally Kellerman. Kellerman was much better suited as a heroine on Star Trek than she was as the butt of jokes in M*A*S*H. If the TOS staff would have allowed it (and GR definitely would not, after the way he gaslighted Majel and had to keep her in the dark about her failure), Kellerman would have been exceptional in a strong, no-nonsense leadership role.
My sense is that Julie Newmar would not have taken well to a job like this, she would have disliked it as a "mannish" part and quit. And I think the sensational Bond Girls Diana Rigg and Honor Blackman were beyond Star Trek's reach.
But again, it was never going to happen because GR had to maintain the fiction that NBC forbad women in command.