I imagine there has to be more than just that random rock edge on their planet, right?
Probably not. After all, the Founders would want to keep the Dominion as far away from them as they possibly can!
The two remote and desolate worlds where we find Founders in "The Search" and "Broken Link" are hideouts, places where the Founders remain unseen and hidden from all Solids, including those they keep at an arm's length through integrating them into the Dominion. The worlds are rather tellingly located on the outer fringes of the Dominion or even outside it (as evidenced by the ease by which our heroes reach those places from the wormhole Gamma end, which is originally located outside the Dominion).
They certainly wouldn't want to build any Jem'Hadar or Vorta there, let alone ships or weapons - all they would really need is a communications room for contacting a select bunch of Vorta in the Dominion proper, and telling them how to run the show. We see some facilities on the first hideout world, namely those used for interrogating the Defiant
captives, and some Vorta there, but there might be very little else on that planet besides the big seas of Founder goo and the few inspirational rock outcroppings sticking out of the seas.