I mean there is no such thing as "white people" in the context in which the term is being used here, one massive group of people unified in their evil scheme to keep the black man down.
There doesn't have to be an overt conspiracy for it to happen. Now, it's an interesting philosophical question as to whether or not someone should be considered morally culpable for being the beneficiary of white privilege, and I'd probably argue against that idea, but I feel very strongly that such privilege does exist, and that it is a problem of our society.
I'd recommend the works of Tim Wise
if you're at all interested in the subject (and, since you're reading this and posting about it, you are).