^I tend to agree. I mean, time definitely passes differently on the island, but I don't think the island itself actually time-travels. It just seems a little too silly.
I think of it this way: The island is out of "alignment" with the regular world's "space/time" flow. The Island is like a radio station broadcast transmitting at a certain frequency. Outside people trying to find the island must first find the correct "frequency" (the coordinates) of the island to see it, just like you have to move the dial and search through lots of frequencies on a radio to find a radio station. But once you have the correct frequency ("coordinates") you can find it every time with out fail. If you are a little off in the frequency you can still hear the radio station it's just fuzzy, and if you are a little off in the coordinates you can still reach the island, just at a different point in time. What Ben did was change the "frequency" of the island so the old coordinates will no longer work to find it. Now any outsiders will have to start searching all over again to find the islands correct "frequency" before they can get there again. At least that is my theory/thinking on how the island moved.