Here's what happened:
Voyager was slipping in the ratings by its third season so word came down from UPN to mix things. Make the show more action based and 'sex it up' to appeal to a wider audience.
It was Brannon Braga who came up with the idea of adding a 'borg babe' (his words not mine) to the cast. UPN and paramount loved the idea but the budget wouldn't allow for 10 regular cast members so one of the characters had to go.
All the actors (with the exception of Kate Mulgrew) had signed 7 year contracts, but like most shows, there's a clause in the contracts that allows the studio to opt out of an actor's contract at the start of a new season. The actors are generally paid out of certain percentage of the remainder of their contract when they're not renewed.
It was originally decided that Garrett Wang. Wang was notorious for being late and unprepared for filming and at one point late during the 3rd Season was suspended from the show for two weeks. Harry Kim was attacked by Species 8472 in Scorpion Part 1 and was going to be killed off in part 2. During the hiatus however, Wang was voted by People Magazine as one of the 50 sexiest people on TV. The producers felt they couldn't let this free publicity go so they decided to keep Wang on cast.
It's largely believed that Jeri Taylor decided to axe the Kes character because she felt that character wasn't working and hard to write for. Several members of the writing staff like Kenneth Biller and Jeo Menosky were not happy with the decision. And Kate Mulgrew was furious.
Originally Kes was going to stay on board for a few episodes. The Gift was supposed to be the 5th episode of the fourth season and Kes' mental abilities were supposed to gradually expand over the previous 4 episodes. However, Jennifer Lien's schedule made her only available for the first two episodes.
Well, Jeri Taylor wasn't telling the truth when she stated that "Kes wasn't working and was hard to write for". In fact, Mrs. Taylor herself managed to come up with two good Kes stories in the books "Mosaic" and "Pathways". Not to mention that Taylor and the other writers had no problems with writing stories for Kes in the first three seasons of the show.
"Jennifer Lien's schedule made her only available for the first two episodes". What schedule? Have I missed something?
As for the ratings, they went up when Seven joined but it didn't take long before they were lower than in the first three seasons.