I agree that for 'cultural' impact, beyond the usual sf fans, it would be HP. That reached way more people and for a lot of years.
LotR had great films (and, of course, great books) but how many references to it do you hear/see, besides Big Bang Theory? It'll always be something special to many, but already forgotten by most who saw the films.
I think so many things have become niche, big niche or small, that having something that attracts the attention of a HUGE, diverse population--and maintains that attention over time--is rarer now.