Neelix will usually get to a reasonable facsimile of the truth in situations where he's providing intel to either puff himself up or that's he's just not sure about. So he says unequivocally, that the ship abductions having been going on for months. I'll give him full marks on that one, especially since, as it happens, the Ocampan nurse verifies that remark a bit later on. Then he starts high and works his way down on the estimate of how many vessels have been so hijacked. He stops at fifty. Now, if one figures how long it takes Banjoman to make his crew assessments on each ship, such an estimate might not be too far off the mark. But for the sake of argument, let's say it's off by half. That's not very much if we don't work under the assumption that ships were all taken from the same area, but instead from all over the three other quadrants, or even just the entirety of the Alpha. As been already pointed out, some races would be unlikely to divulge such losses. So, I think that it's quite possible that knowledge of these goings on would not have been pieced together. As to his contention that he supposes the ships are sent back if their crew is found lacking, I'm not sure why he would even suggest it at all if he didn't have some sense that it was true, perhaps viewing a single instance in which it occurred. Strangely, I find that I haven't able to find that line now, when I was sure I knew when it was delivered previously.
As far as the contention that the debris field was a starship graveyard, it would seem unlikely that Neelix would make the statement above, if he was scavenging an area that obviously contained some of those ships that had been shanghaied. However, having had another look at the scene it would appear that a number of the objects appear to be of the size and configuration, that a conclusion that they are remnants of ships seems pretty reasonable. That they are debris of abducted vessels or just what's left of internecine Kazon conflict is problematic.