Well, Scotty rigged up the transporter in a way that it didn't normally operate in. Any issues in squaring the behavior of the transporter in "Relics" with the transporter behavior in episodes like "Realm of Fear" can, I think, be attributed to Scotty's jury-rigging.
The first time anything like storing people inside the transporter was done was "Day of the Dove". There it was called "suspended in transit." Naturally, they didn't elaborate on what that means, and Scotty's characterization of it as "non-existence" is sufficiently vague to allow for different answers to the question of how much time Kang and his party perceived to have passed.
Of course, if you go back to "Obsession", there are multiple attempts made to rematerialize Kirk and Garrovick, so arguably their patterns were kept in a similar place. Also, the rejoining process in "The Enemy Within" could have had the Kirk(s) in the same place, as well. Is it what was later called the "pattern buffer"? Oh, most likely!