Joss freely admits he has no grasp of real science. But his stories are about people, and the firefly verse is just a framework to let him tell his stories about the people.
Joss wasn't the only worldbuilder involved though; I'd be curious to see how much Tim's input went in. Also, Firefly's universe is, overall, still better constructed than most television sci-fi universes. I mean, there's elements in it that are still more plausible or make more sense within, ironically, science, than say, Babylon 5. (I don't just mean that there's no FTL either.) Also, when Whedon has joked about "don't mention science" to me, it seemed more in the vein of "save me from fanboys quoting physics books". People seemed to take his comment to mean he was ignorant, anti-science, or didn't care one whit about plausibility.
The map of the verse is interesting. While it may seem a bit outrageous, I do believe that within the fiction of Firefly itself, it is stated that the "verse" was considered an incredible find; not just a random star system humanity picked and aimed at. If the verse is suppose to be at the extreme edge of possibility within the setting itself, I find suspension of disbelief easier - after all, we don't complain about Hyperspace in tons of scifi. Meanwhile, "all" we're talking about here is a very rare arrangement of stars.
The scenario also DOES make for good fodder for plot and drama; the weird-ass verse creates a very unusual geopolitical landscape for a setting in which nobody has faster than light travel and is restricted to living in a single stellar neighborhood. Even what we already saw of Firefly did begin to take advantage of this setting.
As for the Reavers, they're actually a much bigger plot hole potentially than the shape of Firefly's star system(s). I suppose, the way I look at it is that the Reavers may be driven insane by the sight/smell of normal humans; the Pax may cause the Reavers to react to stimulation from pheromones as well. Fear response in humans may actually drive them into a greater rage; from their behavior, the Reavers themselves may actually be truly fearless and thus not trigger aggressive in one another.
Also, bear in mind, that in Firefly's universe people are as used to basic spaceship controls as we are to automobiles. People have a lifetime of experience with the technology. Any ships advanced enough to be workaday vessels in such a universe must also have very good computer and automated systems to make /basic/ (not fancy flying) operation relatively simple. It might be plausible that the Reavers remember enough of spaceship operations at a hindbrain level that they can operate airlocks, push buttons on nav screens to travel to charted planets, etc. Keep in mind that there's every sign the Reavers do not maintain their ships. The ships are falling apart.
Many have "core leaks" implying their reactors are on the verge of giving out. The Reavers might logically be headed for extinction within another decade or so, unable to use working ships to get anywhere.