Obviously fictional materials with impossible, fantastical properties would have to have those properties defined either by forces that are unknown to real science, or by known forces manifesting themselves in ways that are unknown to real science, or both. Another possibility that's been alluded to is some effect that masks inertial mass.
Whatever the mechanism, a) it's fictitious, b) it's physically impossible, or else it would be possible to make the substance in the real world, and c) it's just one thing in a long list of physically impossible things, including Mjolnir, regarding which we are tasked to suspend disbelief. Accepting the Infinity Stones but drawing the line at Cap's shield would be completely ridiculous, for example.
There's simply no way that one can attempt to rationalize how these things work that isn't going to run into something objectionable, or otherwise this stuff would be real.
I mean my initial point is that it should just be shown as heavier than it is, wasn't saying it should be taking Caps arm off.

And yes the Stones are the major ridiculous part of it all, I really wanted the Eternals to arrive and clear something up about them, maybe they still will someday.