Automating Fairness Configurations for Machine Learning
Authors: 
Haipei Sun, Yiding Yang, Yanying Li, Huihui Liu, Xinchao Wang, Wendy Hui WangAuthors Info & Claims
Haipei Sun, Yiding Yang, Yanying Li, Huihui Liu, Xinchao Wang, Wendy Hui WangAuthors Info & ClaimsPages 193 - 201
Abstract
Recent years have witnessed substantial efforts devoted to ensuring algorithmic fairness for machine learning (ML), spanning from formalizing fairness metrics to designing fairness-enhancing methods. These efforts lead to numerous possible choices in terms of fairness definitions and fairness-enhancing algorithms. However, finding the best fairness configuration (including both fairness definition and fairness-enhancing algorithms) for a specific ML task is extremely challenging in practice. The large design space of fairness configurations combined with the tremendous cost required for fairness deployment poses a major obstacle to this endeavor. This raises an important issue: can we enable automated fairness configurations for a new ML task on a potentially unseen dataset?
To this point, we design Auto-Fair, a system that provides recommendations of fairness configurations by ranking all fairness configuration candidates based on their evaluations on prior ML tasks. At the core of Auto-Fair lies a meta-learning model that ranks all fairness configuration candidates by utilizing: (1) a set of meta-features that are derived from both datasets and fairness configurations that were used in prior evaluations; and (2) the knowledge accumulated from previous evaluations of fairness configurations on related ML tasks and datasets. The experimental results on 350 different fairness configurations and 1,500 data samples demonstrate the effectiveness of Auto-Fair.
References
[1]
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. 2018. A reductions approach to fair classification. arXiv preprint arXiv:1803.02453(2018).
[2]
Marcin Andrychowicz, Misha Denil, Sergio Gómez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. 2016. Learning to Learn by Gradient Descent by Gradient Descent. In Proceedings of the 30th International Conference on Neural Information Processing Systems. 3988–3996.
[3]
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine Bias: there’s software used across the country to predict future criminals. And it’s biased against blacks. ProPublica 2016.
[4]
Y. Bengio, S. Bengio, and J. Cloutier. 1991. Learning a synaptic learning rule. In IJCNN-91-Seattle International Joint Conference on Neural Networks, Vol. ii. 969 vol.2–.
[5]
Dan Biddle. 2006. Adverse impact and test validation: A practitioner’s guide to valid and defensible employment testing. Gower Publishing, Ltd.
[6]
JC de Borda. 1784. Mémoire sur les élections au scrutin. Histoire de l’Academie Royale des Sciences pour 1781 (Paris, 1784) (1784).
[7]
Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39, 3/4 (1952), 324–345.
[8]
Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. 2009. Building classifiers with independency constraints. In 2009 IEEE International Conference on Data Mining Workshops. IEEE, 13–18.
[9]
Toon Calders and Sicco Verwer. 2010. Three naive Bayes approaches for discrimination-free classification. Data Mining and Knowledge Discovery 21, 2 (2010), 277–292.
[10]
Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. 2017. Optimized pre-processing for discrimination prevention. In Advances in Neural Information Processing Systems. 3992–4001.
[11]
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. 2016. RL$23032$: Fast Reinforcement Learning via Slow Reinforcement Learning. CoRR abs/1611.02779(2016).
[12]
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference. ACM, 214–226.
[13]
Cynthia Dwork, Ravi Kumar, Moni Naor, and Dandapani Sivakumar. 2001. Rank aggregation methods for the web. In Proceedings of the 10th international conference on World Wide Web. 613–622.
[14]
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 259–268.
[15]
Gabriel Goh, Andrew Cotter, Maya Gupta, and Michael P Friedlander. 2016. Satisfying real-world goals with dataset constraints. In Advances in Neural Information Processing Systems. 2415–2423.
[16]
Nina Grgic-Hlaca, Muhammad Bilal Zafar, Krishna P Gummadi, and Adrian Weller. 2016. The case for process fairness in learning: Feature selection for fair decision making. In NIPS Symposium on Machine Learning and the Law, Vol. 1. 2.
[17]
David Ha, Andrew M. Dai, and Quoc V. Le. 2016. HyperNetworks. CoRR abs/1609.09106(2016).
[18]
Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315–3323.
[19]
David R Hunter 2004. MM algorithms for generalized Bradley-Terry models. The annals of statistics 32, 1 (2004), 384–406.
[20]
Faisal Kamiran and Toon Calders. 2009. Classifying without discriminating. In 2009 2nd International Conference on Computer, Control and Communication. IEEE, 1–6.
[21]
Faisal Kamiran and Toon Calders. 2012. Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems 33, 1 (2012), 1–33.
[22]
Faisal Kamiran, Asim Karim, and Xiangliang Zhang. 2012. Decision theory for discrimination-aware classification. In 2012 IEEE 12th International Conference on Data Mining. IEEE, 924–929.
[23]
Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. 2012. Fairness-aware classifier with prejudice remover regularizer. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 35–50.
[24]
Toshihiro Kamishima, Shotaro Akaho, and Jun Sakuma. 2011. Fairness-aware learning through regularization approach. In 2011 IEEE 11th International Conference on Data Mining Workshops. IEEE, 643–650.
[25]
John G Kemeny. 1959. Mathematics without numbers. Daedalus 88, 4 (1959), 577–591.
[26]
Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika 30, 1/2 (1938), 81–93.
[27]
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807(2016).
[28]
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. In Advances in Neural Information Processing Systems. 4066–4076.
[29]
Ke Li and Jitendra Malik. 2016. Learning to Optimize. CoRR abs/1606.01885(2016).
[30]
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2019. A Survey on Bias and Fairness in Machine Learning. arXiv preprint arXiv:1908.09635(2019).
[31]
Cathy O’neil. 2016. Weapons of math destruction: How big data increases inequality and threatens democracy. Broadway Books.
[32]
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. 2017. On fairness and calibration. In Advances in Neural Information Processing Systems. 5680–5689.
[33]
Novi Quadrianto and Viktoriia Sharmanska. 2017. Recycling privileged learning and distribution matching for fairness. In Advances in Neural Information Processing Systems. 677–688.
[34]
J. Schmidhuber. 1992. Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks. Neural Computation 4, 1 (1992), 131–139.
[35]
Till Speicher, Hoda Heidari, Nina Grgic-Hlaca, Krishna P Gummadi, Adish Singla, Adrian Weller, and Muhammad Bilal Zafar. 2018. A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual &Group Unfairness via Inequality Indices. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2239–2248.
[36]
Joaquin Vanschoren. 2018. Meta-learning: A survey. arXiv preprint arXiv:1810.03548(2018).
[37]
Vladimir Vapnik and Akshay Vashist. 2009. A new learning paradigm: Learning using privileged information. Neural networks 22, 5-6 (2009), 544–557.
[38]
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. 2016. Learning to reinforcement learn. CoRR abs/1611.05763(2016).
[39]
Blake Woodworth, Suriya Gunasekar, Mesrob I Ohannessian, and Nathan Srebro. 2017. Learning non-discriminatory predictors. arXiv preprint arXiv:1702.06081(2017).
[40]
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. 2017. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1171–1180.
[41]
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In International Conference on Machine Learning. 325–333.
[42]
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. ACM, 335–340.
- Automating Fairness Configurations for Machine Learning
Recommendations
Airtime Fairness for IEEE 802.11 Multirate Networks
Under a multi rate network scenario, the IEEE 802.11 DCF MAC fails to provide air-time fairness for all competing stations since the protocol is designed for ensuring max-min throughput fairness and the maximum achievable throughput by any station gets ...
Optimal CWmin selection for achieving proportional fairness in multi-rate 802.11e WLANs: test-bed implementation and evaluation
WiNTECH '06: Proceedings of the 1st international workshop on Wireless network testbeds, experimental evaluation & characterizationWe investigate the optimal selection of minimum contention window values to achieve proportional fairness in a multirate IEEE 802.11e test-bed. Unlike other approaches, the proposed model accounts for the contention-based nature of 802.11's MAC layer ...
Achieving MAC-layer fairness in CSMA/CA networks
We demonstrate that CSMA/CA networks, including IEEE 802.11 networks, exhibit severe fairness problem in many scenarios, where some hosts obtain most of the channel's bandwidth while others starve. Most existing solutions require nodes to overhear ...
Comments
Information & Contributors
Information
Published In
April 2021
726 pages
ISBN:9781450383134
DOI:10.1145/3442442
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 03 June 2021
Check for updates
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
WWW '21
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,899 of 8,196 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 134Total Downloads
- Downloads (Last 12 months)12
- Downloads (Last 6 weeks)2
Reflects downloads up to 13 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format