Are you sure the masses are fed by replicated food? Sure, starships and starbases are equipped with them. On Earth and other major worlds it seems either people don't like replicator-food or just enjoy growing and preparing their own food. But on colonies and other 'regular' worlds, it seems a part of the food at least is conventional. Even moreso in the 23rd century, where we're not even 100% sure they have proper replicators.
And last but not least, the Federation actually has a Bureau of Agriculture, and the tribble-episodes show us parts of the process of distributing grain to outer colonies, which includes robot ships and spacestations with grain storage facilities.
Scant evidence from TOS could imply that there are farming worlds that are controlled by a few hundred people, while automated spacecraft visit the planet on a regular basis to distribute the produce among colonies and space stations.