How can we be so sure, though? We really have no idea of who, actually, gets to use a replicator (can literally anybody who wants one, get one? And if so, how do they do that?), what conditions they must satisfy to continue to be allowed to use one, who ultimately controls replicator access, etc.
About the only thing we've really heard on this point is those who refuse to use replicators, such as Picard's family. Robert Picard would not allow replicators in his home and on his property. So it would seem that people are free to NOT use replicators, but we don't know who is free TO use them.
And following on from that, it is theoretically possible that the Federation government really does control replicator privileges, and could set conditions which would be required for their use.
Just spitballin'.