The nature of competition seems to favor the exceptions rather than the rule. There are more lefties in baseball than the general population and obviously basketball players are taller than the general population. The average gymnast is a waif. The average jockey is a midget. The average female runner has an extremely narrow (aka male-like) waist-to-hip ratio compared to the general population. None of this is to disparage the hard work that goes into training but those physical deviances from the norm make a huge difference when it comes to the top 1% of performers. There are exceptions, of course, like Mookie Betts being small for a power-hitter. I find athletes like that more inspiration than those with big genetic/hormonal advantages.
If trans, statistically speaking, have a built-in genetic advantage over women, then it will encourage them to pursue sports in the same way being really tall would encourage someone to pursue basketball or being really small might encourage one to be a jockey. Once that happens, even if the total number of trans in proportion to the general population is tiny, they will start to comprise a larger and larger portion of the athletic demographic. If this happens and they start dominating women's sports, it's going to cause problems.
I mean, there's a reason boxing, for instance, separates fighters based on weight classes. When biological advantages get lopsided enough there's only so much skill can be used to compensate, otherwise it's simply not fair. And need I bring up all the past controversies over iron curtain Olympic athletes as to whether they had two X chromosomes or how much juicing they were doing?
So sure, let things go where they go, but if you think there won't be pushback from some athletes over fairness, you're living in a polyanna dreamworld.