The fate of driverless cars rests in the hands of future juries, who will decide whether an automotive company is <i>liable</i>, to the tune of millions of dollars, for accidents that will inevitably happen and have no other clear explanation.
Recall that the US light plane industry (Cessna, Piper, Beechcraft) was at one point virtually dead because they kept losing massive lawsuits after their customers would crash. In many of those cases they were held liable just because their aircraft's design contributed
to pilot error. I recall one case my former coworkers had defended a claim that even though the crash was caused by carburetor icing (because the pilot forgot to apply carb heat on approach) in an airplane built in the 1930's, the crash wouldn't have happened if the company had later provided an upgrade to a fuel-injected engine. My best friend works as a professional engineering witness in vehicle crashes. If there's a way to blame some flaw in the vehicle and convince a jury that the manufacturer is liable, they will so vote.
Generally the profit on a vehicle is a few thousand dollars, sometimes as low as a few hundred, so losing one multi-million dollar lawsuit can wipe out the gains from building a thousand or more cars. Thus manufacturers take great pains to avoid causing such lawsuits, and really try not to lose the ones they do face.
So even though driverless cars may drop the rate of vehicle fatalities tenfold, what's more important is whether they up the number of successful multi-million dollar damage awards against the manufacturers by a significant degree. To hedge against that, manufacturers will take on much more insurance, adding it to the price of the vehicle, and possibly designing very defensively regarding the technology, such that if all the sensors aren't in top condition (dirty camera lenses, etc), the vehicle might refuse to proceed. Another approach they might take is use all the vehicles sensors to indict the driver ("and here she is, sipping a slurpy and reading the paper through a residential neighborhood filled with small children").