Driverless cars scare the shit out of me. Technology fails sometimes, and when that happens, humans need to be able to step in and take control. Drivers already don't pay enough attention to the road! I can only imagine how much less they'll be paying attention when they're even not responsible for driving.
You do realize it's not hard to have safety features that include, "if everything goes pear-shaped, pull to the side of the road and slow to a stop," right?
Real life is not Star Trek. Your dashboard isn't going to yell "ERROR! ERROR!", throw sparks in your face, and fly off a cliff because it ran over an unexpected rock.