It's true that Sci Fi was used by some as a pejorative, but that wasn't everybody. When I was a kid-- the 70s or so-- we used Sci Fi and Sciffy more as terms of endearment, and we considered anybody who looked down their nose at it (and B-Movies and Pulps) as being way too uptight. And counter-productive. The same way that we look down upon the politically correct these days.
As for common usage, the common people use "science fiction" itself as a pejorative. I remember one time when I was hanging around with some security guys at a hospital where I worked, the director was mocking one of the guys for liking "all that science fiction stuff." Turns out he was a Stephen King fan.
But that wasn't what I was talking about. I was talking about usage of the term to include things that are by no stretch of the imagination related to SF, like Horror and Fantasy. Labels should be useful and instructive. Should we consider Three's Company
Sci Fi because it takes place in an alternate universe?