My own experience: When I've worked in groups charged with deliberately considering diversity as part of some some decision-making process (or in forming the group in the first place), I've always felt the result has been more thoughtful, of higher quality, and led to more confidence in the outcome than it otherwise would have. Every single time.
Diversity has value. Racists want you to apologize for it or back down from it, but they're not just being hateful, they're weakening institutions.