@Eve I hear you. TERFs like that line of thinking, but so do a lot of well-meaning socially liberal people.
I think the internal concept of gender is so invisible and the reality of sexism are so visible to cis people, and it's easy for them to think we'd be better off without gender.
My main response would be that there's a history of trans people in different cultures, with medically documented distress, and improvements from transitioning to show that gender is very real and important to some people. And I'd wager a lot of cis people would be distressed if forced into the wrong gender roles or anatomy.