The show's failure to more extensively flesh out and establish a broader stable of recurring characters isn't a continuity issue; it's an issue of missed opportunity.
I am recalling an interview in which tptb stated (early in the show's run) that they were being very deliberate when writing stories that involved introducing crew members, because once into'ed, they'd be stuck with having to deal with them, having to explain their absence or ignored altogether.
I don't remember the magazine where I read that, but, the lack of an extended community onboard wasn't a failing per se, but a deliberate choice they made. And an unfortunate one IMO, as the main cast gelled so thoroughly, so quickly, that we, the audience, needed
to have them interact with others more than ever, to avoid getting bored by them.
Voyager was made smaller than a Galaxy class ship, in order to force the crew to be in each other's faces, but the reality is that it wound up not really mattering what size the ship was. Everyone had their own private, spacious quarters , even newcomers Neelix and Kes. They weren't tripping over each other, or in each others' faces...I wish there had been a larger number of people onboard, so that the smallness of the ship could have been conveyed more overtly.
Oh well. Hindsight is always 20-20.