Except that these books, even the bloated ones, are full
of seemingly throwaway references and moments that turn out to be hinting at future reveals large and small (I don't think and didn't suggest that this one would be any of great import). Is this one of them? Possibly not. But I'm not sure Martin would specifically highlight a bit of worldbuilding that looks like contrivance without having a reason for doing so. And since the series seems to be coming down to an opposition between the magic of the Others and the magic of dragons, I don't find it silly to speculate on the significance of the masters of the latter declining to extend their reach into lands troubled by the former.