Driverless cars scare the shit out of me. Technology fails sometimes, and when that happens, humans need to be able to step in and take control. Drivers already don't pay enough attention to the road! I can only imagine how much less they'll be paying attention when they're even not responsible for driving.
In 2010 there were an estimated 5 + million with a capital M car accidents of which ~ 32,000 people were killed in the US. This makes cars IMO the biggest serial killer in America and the people who kill who are driving them.
If automation would even half that number - and I believe it could do even much better than that - you'd think the public would overwhemingly support the driverless car and the technology necessary to run it - occassional failure or not.
But like I said earlier in this thread - privacy rights groups I'd imagine will be one of the biggest hurdles to a large scale use of any such tech.