And age, age, age. I saw a film called Trilogy of Terror when I was pretty young on TV. To this day, I cannot rewatch it because of the Zuni Devil Fetish Doll segment. I wouldn't sleep for at least a week after that.
I was terrified by
Ghostbusters when I first saw it as a kid (I think I was eight). Slimer in particular.
But OTOH it's not just age. The old man was in his thirties when he first saw
The Exorcist and he still couldn't handle it. (He didn't need quite the number of tries to get through it that
martok describes in his case, but OTOH maybe he would have been better served by taking it in stages.

)
The Exorcist is of course legitimately scary and deserves the hype, but I couldn't say it ever got in my head to that extent. Honestly
The Texas Chainsaw Massacre (the real one),
The Poltergeist and
The Blair Witch Project all stuck with me just as much or more.
Cannibal Holocaust much moreso.
But honestly the most horrifying films aren't even necessarily "horror" films, per se. Often they're war films, which are about the inescapable horror of the real rather than about jump scares and the supernatural. Russian war films especially: like
Come And See or
The Chekist. (The latter, with its repeated images of firing squads and bodies being dumped, burned itself into my brain like almost no other movie ever has with the possible exception of
Schindler's List.)