Going by the episodes where characters specifically mention how many people are onboard (in bold), it never makes sense.
The 37's (S2E01) - 152 crewmembers (excluding Kes, Neelix and the Doctor)
9 deaths - Alliances, Meld, Investigations, Innocence, Basics Pt 2, Unity, Latent Image (Ensign Jetal)
1 Addition - Naomi Widman
Net loss between The 37s and Distant Origin = 8 crewmembers
152-8 = 144 crewmen + Kes and Neelix = 146 people aboard
In Distant Origin (S3E23) there are 147[lifesigns detected, so who knows where the additional lifesign came from
7 deaths - Scientific Method, Ashes to Ashes (Lindsay Ballard), Living Witness, One
Net loss between Distant Origin and In the Flesh is 7, 146-7 = 139
In the Flesh - The doctor says there are 127 people aboard which would imply that 12 have died offscreen since Distant Origin.
Next mention of the number of crew is in Gravity (S5E13)when Tuvok mentions there are 152 crewmembers. So suddenly the crew count has jumped up to the number mentioned in the 37s. He isn't talking about Voyager's original crew complement as that was 141.
2 episodes in later in Dark Frontier (S5E15), the Borg detect 143 lifesigns on board the ship which contradicts both Gravity by being drastically lower in a very short space of time, In the Flesh by being much higher, and even "Distant Origin by having only 4 less lifesigns detected since then despite 7 deaths having occurred since then. Although if we're in the mood for fanwankery, Distant Origin and Dark Frontier are the easiest to reconcile if we assume that some of the dead crewmembers were miraculously revived.
But then in Someone to Watch Over Me (S5E22) it is mentioned there is a crew of 146, so Voyager has gained 2 crewmembers since Dark Frontier (not 3 since the Doctor wouldnt have been included in a lifesign scan but Neelix would have included him)
3 Deaths - Equinox Pt 2, Friendship One
6 additions - Icheb (Collective) and the Equinox crewmembers
Net addition - 3 between Someone to Watch Over Me and Author, Author
So the final mention is in Author, Author (S6E20) - 147 people on board. Counting from S5E22, there should now be 149 aboard but it isn't a stretch to believe that 3 people have died offscreen....
Overall though, when taking into account what I've written above and that huge casualty list seen in "Imperfection" full of the names of people we hadn't seen die and its obvious that the writers simply didn't care about keeping continuity with the numbers. And frankly, most people who've watched and enjoy Voyager don't care anywway.