Well the producers claim that the Kes character wasn't working and they didn't know what to do for her, but as sfdebris said in his "One" review - the writers didn't know what to do with ANYONE so it was hardly a special case for Kes. And Kenneth Biller even said in the season 4 extras that it was a pity that Kes left because there was a lot of potential for storylines given her unique physiology and telepathic powers.
SFdebris summed up the Kes/Seven exchange in his Kes retrospective like this
"Overall while you could have replaced Harry with just about anything and not lose, replacing Kes with Seven was like replacing a home theater system with a motorcycle. You haven't actually upgraded, you've just wound up replacing something with something different. You have something new and great but you've also lost something else you've had that you really liked".
Personally I thought getting rid of Kes was a big mistake, her character had a lot of potential and keeping Kes and Seven on the show would have helped make Voyager be more distinctive by not only having it be much more female-centric but also by having a more diverse mix of characters.