Okay, so I’m Asian, and suppose I’m with two other people, A and B, of which at least A is White.
Suppose A says “Wow, there’s lots of Asians around here”. Then B stares at A and subsequently tells A that is racist.
If you’re A, you have not done anything that will cause me to want to distance myself from you. You were just remarking that there seems to be a high quantity of Asians in the vicinity. It’s not like you’re saying or even implying something prejudiced about Asians. I saw no racism in that statement, and I wouldn’t even have contemplated the possibility until B decided to remark on it.
If you’re B, then you have signaled to me that you heard that utterance and saw racism in it where I still do not see racism after your remark. You have somewhat persuaded me that I should distance myself from you, because if you heard that and decided there’s malice in it, then I don’t want to be around you, because even though I’m in the demographic you’re happening to believe to protect at the present, I’m sure somewhere down the road I’m bound to make a statement I think is innocent but that you will judge me as malicious for, given that you saw this in that statement.