Again, he's The Joker. It's doubtful the things he says has any reason. There may have been other lines in the movie suggesting that these were some of the criminals taken down in the big mob bust but, again, they could be anyone from the Bosses down to the soldiers meaning they range from the criminal overlords down to the un-made guy who gets stuck roughing up the local union rep for the protection money. In fact, it's heavily suggested in the movie that Harvey and the Mayor knew the bosses and upper-level guys would make bail and be back on the streets while all of the lower-level guys would be stuck serving their sentences/going to trial. But it certainly seemed to be a bit of a mix of various criminals, I doubt the Ving Rhames criminal was meant to be anyone involved with the mob if from the "moral choice" he made alone.