Fairness Increases Adversarial Vulnerability. (arXiv:2211.11835v2 [cs.LG] UPDATED)

The remarkable performance of deep learning models and their applications in
consequential domains (e.g., facial recognition) introduces important
challenges at the intersection of equity and security. Fairness and robustness
are two desired notions often required in learning models. Fairness ensures
that models do not disproportionately harm (or benefit) some groups over
others, while robustness measures the models’ resilience against small input

This paper shows the existence of a dichotomy between fairness and
robustness, and analyzes when achieving fairness decreases the model robustness
to adversarial samples. The reported analysis sheds light on the factors
causing such contrasting behavior, suggesting that distance to the decision
boundary across groups as a key explainer for this behavior. Extensive
experiments on non-linear models and different architectures validate the
theoretical findings in multiple vision domains. Finally, the paper proposes a
simple, yet effective, solution to construct models achieving good tradeoffs
between fairness and robustness.