Time probably flows different in the other worlds.
Their connection with us could be many things.
They could be a product of our collective subconscious. Our tellings, believing and retellings of the legends, myths and fables actually brought them into existence. Or perhaps the worlds of fantasy are linked to us via dreams. (an idea actually touched upon in the series) So the fairy tales are our interpretations of these"dreams". Obviously sometimes things get lost in the "translation".