Here we cover introductory resources for each trustworthy ml field. This includes tutorials, papers, talks and course materials. While we try our best to ensure that this list of resources is up-to-date and comprehensive, it gets hard to keep up with all the great work out there. Did we miss a useful reference or an important paper? Email us!

general

tutorials & talks

• Nicolas Papernot. “What does it mean for ML to be trustworthy?”
• Himabindu Lakkaraju. “Machine Learning for High Stakes Decision Making: Challenges and Opportunities.”
• Timnit Gebru and Emily Denton. “Tutorial on Fairness Accountability Transparency and Ethics in Computer Vision.” CVPR, 2020.

courses

• Nicolas Papernot. “Trustworthy Machine Learning.” University of Toronto.
• Trevor Darrell, Dawn Song, and Jacob Steinhardt. “Trustworthy Deep Learning.” University of California, Berkeley.
• Piotr Mardziel. “Security and Fairness of Deep Learning.” Carnegie Mellon University.
• Kamalika Chaudhuri. “Topics in Trustworthy Machine Learning.” University of California, San Diego.

books

• Michael Kearns and Aaron Roth. “The Ethical Algorithm: The Science of Socially Aware Algorithm Design.”
• Kush R. Varshney.“Trustworthy Machine Learning.”

articles

• Kush Varshney. “Trustworthy Machine Learning and Artificial Intelligence.” XRDS: Crossroads, 2020.
• Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané. “Concrete Problems in AI Safety.”

interpretability and explainability

tutorials & talks

• Finale Doshi. “A Roadmap for the Rigorous Science of Interpretability.”
• Been Kim. “Intepretability - now what?.”
• Zachary Lipton. “Interpretability: of what, for whom, why, and how?.”

courses

• Himabindu Lakkaraju. “Interpretability and Explainability in Machine Learning.” Harvard University.

books

• Christoph Molnar. “Interpretable Machine Learning - A Guide for Making Black Box Models Explainable.”

articles

• Adrian Weller. “Transparency: Motivations and Challenges.”
• Finale Doshi-Velez and Been Kim. “Towards a Rigorous Science of Interpretable Machine Learning.”
• Zachary Lipton. “The Mythos of Model Interpretability.”
• Cynthia Rudin. “Stop Explaining Black Box Machine Learning Models for High Stakes Decisions.” Nature Machine Intelligence, 2019.

fairness

tutorials & talks

• Arvind Narayanan. “Tutorial: 21 fairness definitions and their politics.”
• Solon Barocas and Moritz Hardt. “Tutorial on Fairness in Machine Learning.” NeurIPS, 2017.
• Sarah Bird, Ben Hutchinson, Sahin Geyik, Krishnaram Kenthapadi, Emre Kiciman, Margaret Mitchell, and Mehrnoosh Sameki. “Fairness-Aware Machine Learning: Practical Challenges and Lessons Learned.” KDD, 2019.

courses

• Moritz Hardt. “CS 294: Fairness in Machine Learning.” University of California, Berkeley.
• Arvind Narayanan. “Fairness in Machine Learning.” Princeton University.

books

• Solon Barocas, Moritz Hardt, and Arvind Narayanan. “Fairness and machine learning: Limitations and Opportunities.”

articles

• Sam Corbett-Davies and Sharad Goel. “The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning.”
• Alexandra Chouldechova and Aaron Roth. “The Frontiers of Fairness in Machine Learning.”
• Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. “A Survey on Bias and Fairness in Machine Learning.”
• Megnan Du, Fan Yang, Na Zou, and Xia Hu. “Fairness in Deep Learning: A Computational Perspective.”

adversarial machine learning

tutorials & talks

• Zico Kolter and Aleksander Madry. “Adversarial Robustness: Theory and Practice.” NeurIPS, 2018.
• Ian Goodfellow. “Adversarial Examples and Adversarial Training.” Stanford University.
• Aleksander Mądry and Ludwig Schmidt. “A Brief Introduction to Adversarial Examples.”
• Ian Goodfellow, Nicolas Papernot, Sandy Huang, Rocky Duan, Pieter Abbeel, and Jack Clark. “Attacking Machine Learning with Adversarial Examples.”
• Bo Li, Dawn Song, and Yevgeniy Vorobeychik. “Adversarial Machine Learning Tutorial.” AAAI, 2018.
• Nicholas Carlini. “Adversarial Machine Learning Reading List.”

articles

• Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. “Adversarial Attacks and Defences: A Survey.”
• Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. “Adversarial Examples: Attacks and Defenses for Deep Learning.”

privacy

tutorials & talks

• Katrina Ligett. “Differential Privacy: The Tools, The Results, and The Frontier.” NeurIPS, 2014.
• Kamalika Chaudhuri and Anand Sarwate. “Differentially Private Machine Learning: Theory, Algorithms, and Applications.” NeurIPS, 2017.
• Katrina Ligett, Kobbi Nissim, Vitaly Shmatikov, Adam Smith, and Jon Ullman. “Differential Privacy: From Theory to Practice.” The 7th BIU Winter School on Cryptography, 2017.
• Katrina Ligett. “Tutorial on Differential Privacy.” Big Data and Differential Privacy, 2013.
• Damien Desfontaines. “A reading list on Differential Privacy.”

courses

• Jonathan Ullman. “Rigorous Approaches to Data Privacy.” Northeastern University.
• Ashwin Machanavajjhala. “Design of Stable Algorithms for Privacy and Learning.” Duke University.
• Aaron Roth. “Differential Privacy in Game Theory and Mechanism Design.” University of Pennsylvania.
• Salil Vadhan. “Mathematical Approaches to Data Privacy.” Harvard University.
• Gautam Kamath. “Algorithms for Private Data Analysis.” University of Waterloo.
• Moni Naor. “Foundations of Privacy.” Weizmann Institute of Science.

books

• Cynthia Dwork and Aaron Roth. “The Algorithmic Foundations of Differential Privacy.”
• Salil Vadhan. “The Complexity of Differential Privacy.”

articles

• Gautam Kamath and Jonathan Ullman. “A Primer on Private Statistics.”
• Kobbi Nissim, Thomas Steinke, Alexandra Wood, Mark Bun, Marco Gaboardi, David R. O’Brien, and Salil Vadhan. “Differential Privacy: A Primer for a Non-Technical Audience.”
• Cynthia Dwork, Adam Smith, Thomas Steinke, and Jonathan Ullman. “Exposed! A Survey of Attacks on Private Data.” Annual Review of Statistics and Its Application, 2017.

other forums

• DifferentialPrivacy.org.

causality

tutorials & talks

• Ferenc Huszar. “Causal Inference in Everyday Machine Learning.” MLSS, 2019. (3 parts)
• Amit Sharma and Emre Kiciman. “Tutorial on Causal Inference and Counterfactual Reasoning.” ACM KDD, 2018.
• Jose Ramon Zubizarreta and Sharon-Lise Normand. “Introduction to Causal Inference.” HDSI, 2019.
• Susan Athey. “Machine Learning and Causal Inference for Policy Evaluation.” Harvard CMSA Big Data Conference, 2015.

courses

• Jonas Peters. “Lectures on Causality.” MIT. (4 parts)
• Robert Ness. “Causality in Machine Learning.” Northeastern University.
• Elena Zheleva. “Causal Inference and Learning.” University of Illinois, Chicago.

books

• Judea Pearl and Dana Mackenzie. “The Book of Why: The New Science of Cause and Effect.”
• Judea Pearl, Madelyn Glymour, and Nicholas Jewell. “Causal Inference in Statistics: A Primer.”
• Guido Imbens and Donald Rubin. “Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction.”
• Hernán MA and Robins JM. “Causal Inference: What If.” Boca Raton: Chapman & Hall/CRC.
• Emre Kiciman and Amit Sharma. “Causal Reasoning: Fundamentals and Machine Learning Applications.”

articles

• Donald Rubin. “Causal Inference Using Potential Outcomes: Design, Modeling, Decisions.” American Statistical Association, 2005.
• Judea Pearl. “An Introduction to Causal Inference.” The International Journal of Biostatistics, 2010.
• Judea Pearl. “The Seven Pillars of Causal Reasoning with Reflections on Machine Learning.” Communications of the ACM, 2019.