I do think that Denise made a hasty decision to leave, but I can also understand why. The writers had no idea how to write ANY of the female characters, and were still struggling with it in season seven. But throw in a female character in what was traditionally a male role while also trying to have it played as 'this is an everyday kind of thing, it's not questioned' and the writers of the time were probably trying to figure 'how do we have an episode about Tasha that doesn't focus on a woman trying to do a man's job and doesn't come across as sexist?' Like I said, they had enough trouble with the female characters in the more maternal roles, that doesn't speak as well to writing a female character in a 'protector' oriented role. Denise probably figured that it wasn't going to get any better and, rather than just become scenery, she asked out.
And on top of the above, and as a warning, this is where I start ranting, there was the 'our characters DON'T' clause put in by Roddenberry that I've harped on a couple of times. Tasha's background was richer than all of the other Enterprise crews combined and could have been a goldmine... Except Gene Roddenberry said that 'humans no longer do X, these characters can't do Y,' etc. Symbiosis, for example, could have been a powerful Tasha episode where she comes face to face with her own prior use of drugs while on her homeworld, but because these characters were supposed to be 'perfect,' apparently, pre-Starfleet Tasha was supposed to be a paragon of virtue who, despite having nothing, wanting everything, fighting for her very survival, never turned to drugs. Her infamous 'drugs are bad, m'kay?' speech to Wesley has a subtext that she once was an addict, but because these characters had to be 'perfect,' she couldn't come out and say it. It's that clause, that 'make the characters perfect' requirement that was put into place, that seriously hurt Tasha as a character because they couldn't dig into her background too seriously without violating it.
Looking back on it now, sure, we can come up with a million different ways to have taken the character. But hindsight is 20/20. Now, there are a ton of various 'action female' characters to draw from who could be an inspiration into how to write a Tasha Yar character. These characters did not exist twenty years ago.