That's really a simplified explanation. The Federation didn't cause the Dominion to "fall" all they really did was check their invasion of the Alpha Quadrant, and only managed to do that because the chokepoint that was the wormhole was so often closed that they couldn't bring their full forces to bear.
That one 8472 the Hirogen hunted was described as being wounded even before the hunt started. So isolating one wounded 8472 is not the equivalent of saying Hirogen weaponry can unequivocally defeat them. Being Voyager beat the Hirogen on a number of fights, that would argue against super armor and more for cunning on the part of the Hirogen.
Furthermore the Dominion's polaron weaponry did cut through the shields of the Federation when they first encountered them. It took the Federation years to counter that, Weyoun being surprised when the shields of DS9 held out against their opening barrage. So, it's just as likely those polaron beams would chew up the Hirogen until they adapt as it were. As for their stealth mode? The Dominion detected the Defiant's cloak rather easily with an anti-proton beam. Seems the same concept.
But all that aside, it does come done to numbers. The Dominion regularly fielded fleets of thousands of ships and can build the quickly and breed crews for them even more quickly. An unusually large Hirogen group was a few dozen ships. And given the way they interact with each other, each individual hunter trying to be the alpha, they aren't team players at all.