The problem with Pottersville was mainly that Potter had a monopoly on housing, and therefore could demand high rent for shabby appartments and houses. There might also have been a bigger problem with racism in Pottersville. Potter not only looking down on the poor and the working class when he talks about the "riff raff", he also talks about "garlic eaters", which was a slur against Italians, represented in the movie through Mr. Martini. While Martini not only gets to own a house in Bedford Falls, but run his own bar, he is nowhere to be found in Pottersville.
So, while Pottersville may have had more night clubs, Bedford Falls was a place of more social and economic equality.