I'm not an American. In my country it's definetely not an elitist pastime. It's taught in primary school and at public pools there are usually many people with a migration background.
Yes, I think it's a very American thing. Here it's not taught in schools, but almost everybody learns to swim at a young age, usually by their parents or taking a few swimming lessons. Going to the pool is neither elitist or posh. Actually, it's a very common thing: it's cheaper than going to the gym, for example.
Beside that, Italy is surrounded by the sea from 3 sides and we have a load of lakes and rivers, so it's exceptionally easy to find a place to dive in. I'm a reasonably capable swimmer, and most people I know are at least able to keep afloat easily.
Now, I have noticed that there is some disparity in the reports of accidental drownings of recent immigrants (especially from Africa and the Middle East), compared to nationals and long-time residents (say, Polish or Romanian people), so I'm afraid that maybe we are going in the same direction of the US in this instance, but I hope the second-generation immigrants will avoid this problem.