The music that particularly killed scenes for me (I think it happened twice) was when the music was really high pitched violins that got higher and higher and then it peaked and stayed there forever. I hated when it hit the top and didn't drop back down.
If that's the music I'm thinking of, then that's the "Joker theme".
My reaction was quite different from yours: I found it really added to the scenes in which it occurred. But then, I'm a fan of modernist classical music, so I enjoy that sort of thing.