I completely agree. On one level, it's true that I can't know firsthand what it's like to be someone else - even if I find someone who has literally every single trait in common with me, I can't physically see with their eyes or hear with their ears or think with their brain. The best I can do is make an imaginative leap of empathy, and honestly that's good enough. (Hell, strictly speaking I can't *know* what it was like to be myself ten years ago - I can only know what it's like to be myself today looking at memories from ten years ago.) And that remains the case regardless of trans vs. cis or any other human trait - I believe that it's possible and worthwhile to develop an understanding of other people, and I believe that in general we all have more in common than not (including in ways that someone might assume to be identity-specific).
But when you get caught up in the sound and fury of the culture war, it's easy to forget that. I'd rather we all assert differences and dichotomies less rigidly - I doubt that there's any essence of transness, or any other category, that exists inherently and objectively and makes the categories "real." Sometimes labels are useful to have, but all they can be is contingently useful - and in my experience, as often as not they get in the way and confuse more than they illuminate.