I've never understood this argument myself. Science fiction is just as valid an emphasis as science fiction, and just because it's fiction doesn't mean there shouldn't be rules. If Picard suddenly pulled out a wand and revealed that he'd been a Ravenclaw prefect at Hogwarts prior to attending Starfleet Academy and although his charms work was rusty he was still reasonably good at transfigurations it would certainly be "anything goes", but it would absolutely violate the in-universe rules of Star Trek.
People fixate too much on rules. “Is it entertaining?”, is the only thing I’m worried about.