True, for a long time it was considered pejorative by those within the prose SF field. But it seems to have lost that stigma more recently as the genre has gained more mainstream acceptance. I'm referring more to my perception of how people in the mainstream have used it over the decades. The general public, network and studio execs, video stores and bookstores, etc. have long had a habit of assigning the label "sci-fi" to encompass all speculative fiction, including SF, fantasy, horror, superheroes, and whatnot. And as I said, there are countless mass-media works that blur the line between SF and fantasy, like Star Wars
(which is a sword-and-sorcery fairy tale dressed up with outer-space trappings) or superhero comics (Superman is an alien from another planet, but he's vulnerable to magic and he hangs out with an Amazon princess molded from clay and imbued with life by the Greek gods).
I used to share the view that "sci-fi" was a derogatory label and resisted its use, but I've come to see it from more a descriptivist point of view -- acknowledging how people in general actually use the term, rather than trying to prescribe a single view of how it "should" be (or not be) used.