You're placing entirely too much importance on particular categorizations, "comfort" has much more to do with "what you're used to", than the idea that people are constantly fretting about how population densities in their environment match up to their own self-identification.

As a bi-racial male of the darker persuasion (which in America is basically "not-white"), I find myself much more comfortable around predominantly "white" groups because that has been the most common environment for myself throughout my life.