I'll ask you. If they required something with no business justification, such as firing you for speeding, would you be ok with that too?
I would not, as my driving record is irrelevant to my job.
However, I think drug use can be completely relevant, regardless of the work you're doing. If a company doesn't want to hire people who use drugs, I think that should be their right. If a company doesn't want to hire felons, that should be their right. Hell, if a company doesn't want to hire people with a lot of parking tickets, that should be their right, too.
So parking tickets but not speeding tickets? I'm not exactly sure your distinction here is. It seems that you're disagreeing with your initial point.
Drugs can impact the way a person does their job, but not all drugs and only if they use them on the job. There are better remedies than drug testing unless it's the kind of job that requires constant vigilance (and most don't - certainly not office jobs).
Personally, I think it's shitty the way convicted felons get screwed out of jobs and essentially have their lives ruined for a single mistake (especially since it makes rehabilitation for those who want to reform twice as difficult), but that's an entirely different issue. At least that requires due process and a criminal conviction.