We have seen Jem'Hadar essentially "wrestle" with human heroes in "To the Death" (as they parry with their two-handed bladed weapons and use simple muscular strength against that of their human opponents). They do not appear to be any stronger than humans there.
Similarly, they appear evenly matched with Worf in terms of sheer strength in "By Inferno's Light", and we have seen that Klingons do not possess a strength advantage over our human heroes.
Additionally, they are quite susceptible to punches in the jaw or gut, just as susceptible as the human heroes - an issue unrelated to muscular strength, but interesting nevertheless.
The bodily advantages of the Jem'Hadar thus seem to lie in their invisibility (before engaging in combat); in their mass-produceability; and in their endurance on minimal supplies (provided Ketracel White is available). Their average strength matches that of our heroes, who admittedly probably always represent above
average strength for humans.