Of course, in most of these cases we're talking about people who were actors first and then trained to do the necessary stunts. Not a lot of people start out as stunt performers and then become actors, though it's been known to happen (e.g. Jock Mahoney, Doug McClure, Ann Robinson, Zoe Bell, and of course the ultimate stuntman, Jackie Chan).
Casting a champion swimmer as Aquaman strikes me as overly literal, too, but, to be fair, there is an historical precedent for a Gold-Medal swimmer becoming a movie star.
Anybody remember Johnny Weismuller?