My guess is the flashbacks on the island are being shot in groups. Maybe not all at once. But multiple episodes at a time.
For a number reasons. Mostly costs. Some of the actors in those scenes barely appear in each episode they are in. Plus the cost of having to set up the location shooting over and over again would be huge.