There is no reason to selectively care about it being unfair. The controversy only exists in the first place because it is a great way for republicans and right wing figureheads to have queer people turned into an existential threat to rally support and perhaps distract from issues more materially relevant to voters. Voters that would never care about fairness in women's sports if it didn't involve queer people to hate. You simply have to acknowledge this. Even if you were not the intended audience and now have an interest in the theoretical question through proximity, the number of people that are actually personally impacted by this is near zero and all the attention originates in anti-LGBTQ rhetoric.
I refuse to acknowledge this as controvercy that requires this kind of addressing.
Voters that would never care about fairness in women's sports if it didn't involve queer people to hate.
That's certainly a large percentage of the motivation, granted. I do think a salient point is that there's nobody lobbying to make it unfair in some other way. If there were groups successfully lobbying to, say, abolish weight classes or allow the use of some expensive performance enhancement like a heads-up display, people would also oppose it.
I refuse to acknowledge this as controvercy that requires this kind of addressing.