I took the position that all fiction is genre fiction, because contemporary realism, historical realism, magical realism, and so forth are themselves genres, and because if an opus somehow manages the rare achievement of failing to fit into any existing genre, then it simply becomes the holotype for a new genre.
I'm fond of the argument that mainstream fiction is just a subset of speculative fiction, limiting itself to an approximation of our world in the present or recent past. I mean, even mainstream fiction has speculative elements, imagining people, places, and events that don't exist. The West Wing, for example, posited several imaginary US Presidents, invented imaginary countries, and offset the presidential election cycle from real life by two years. What is that if not alternate-world speculative fiction?
And most any mainstream action movie will have all sorts of physical impossibilities or improbabilities, like people being blown through the air by gunshots without the shooters feeling an equal and opposite reaction, or being blown through the air by explosions and walking away instead of being instantly killed by the concussive shock and shrapnel, or suffering repeated knockout blows to the head without suffering concussion, or getting shot in the shoulder without any significant impairment to their range of motion, etc. A lot of it is pure fantasy, as much as it would be if the action hero whipped out a magic wand or turned out to be half-elf. Plenty of fiction set in the "real world" is considerably more fanciful than a hard science fiction novel set on an alien world in the distant future but sticking faithfully to known physical law.
(I've been known to jump on straight lines that even Groucho would have left alone.)