Elamur Research⌕
☰
home / papers / 2020 / UNQOVERing Stereotyping Biases via Underspecified Questions

UNQOVERing Stereotyping Biases via Underspecified Questions

Summary

Stereotyping biases are known to exist in language embeddings, but how they carry over into downstream question answering (QA) systems had not been studied. The authors propose UNQOVER, a framework that probes QA models and masked LMs with underspecified questions. Each template-generated paragraph mentions two subjects from protected groups, and the question asks about an attribute the context gives no evidence for. Any consistent preference for one subject therefore reflects an association learned by the model.

The authors find that raw model scores are misleading because of two reasoning errors. Answers shift with the order in which subjects appear (positional dependence), and they often stay the same when the attribute is negated (question independence). They design a metric that cancels out both effects. They then apply it to gender, nationality, ethnicity and religion across 15 transformer models. All models show clear stereotyping biases. Bigger models tend to be more biased, and fine-tuning changes bias in ways that depend on both the dataset and the model size. The work offers a reusable tool for evaluating, and eventually mitigating, bias in QA.

Key points

  • Five architectures were probed: RoBERTa in its base and large sizes, and BERT in its distilled, base and large versions. Each was tested fine-tuned on SQuAD, fine-tuned on NewsQA, and as a plain masked LM, for 15 models in total. Code is released at github.com/allenai/unqover.
  • Fine-tuning effects:
    • Fine-tuning on SQuAD tended to raise bias, while NewsQA tended to lower it.
    • Fine-tuning reduced bias in the distilled model but could amplify it in larger models.
  • Nationality results:
    • Rankings show a clear split between Western and non-Western countries.
    • Negative attributes are disproportionately tied to non-Western nations.
  • Ethnicity and religion results:
    • Arab and African-American subjects rank as most negatively associated, and European as least.
    • Muslim ranks most negative among religions, though the overall intensity is small (|γ(x)| ≤ 0.03).
  • The reasoning errors themselves are substantial:
    • RoBERTa models show more positional error than BERT models of similar size.
    • Neither QA models nor masked LMs handle attribute negation reliably.
    • Gender is treated as binary, and the attributes are partly drawn from StereoSet, which may reflect a Western perspective.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.311

@inproceedings{li2020unqovering,
  title = {{UNQOVERing Stereotyping Biases via Underspecified Questions}},
  author = {Li, Tao and Khashabi, Daniel and Khot, Tushar and Sabharwal, Ashish and Srikumar, Vivek},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.311},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.311}
}

References

36 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohsen Abbasi, Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2019. Fairness in Representation: Quantifying Stereotyping as a Representational Harm. In Proceedings of the 2019 SIAM International Conference on Data Mining. find
  2. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find our card
  3. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016a. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. In Advances in Neural Information Processing Systems. find our card
  4. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016b. Quantifying and Reducing Bias in Word Embeddings. In International Conference on Machine Learning Workshop on \#Data4Good. find
  5. Shikha Bordia and Samuel Bowman. 2019. Identifying and Reducing Gender Bias in Word-Level Language Models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop. find our card
  6. Kaytlin Chaloner and Alfredo Maldonado. 2019. Measuring gender bias in word embeddings across domains and discovering new gender bias word categories. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  8. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences. find
  9. Joseph Henrich, Steven J Heine, and Ara Norenzayan. 2010. Most people are not WEIRD. Nature. find
  10. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Quantifying Social Biases in Contextual Word Representations. In 1st ACL Workshop on Gender Bias for Natural Language Processing. find
  11. Brian Larson. 2017. Gender as a Variable in Natural-Language Processing: Ethical Considerations. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing. find
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar S. Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv. find
  13. Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pre-trained language models. arXiv. find
  14. Won Ik Cho, Ji Won Kim, Seok Min Kim, and Nam Soo Kim. 2019. On measuring gender bias in translation of gender-neutral pronouns. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing. find
  15. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. find
  16. Kate Crawford. 2017. The Trouble with Bias. In Conference on Neural Information Processing Systems, invited speaker. find
  17. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol… find
  18. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Acco… find
  19. Sunipa Dev, Tao Li, Jeff Philips, and Vivek Srikumar. 2020a. On Measuring and Mitigating Biased Inferences of Word Embeddings. In the Thirty-Fourth AAAI Conference on Artificial Intelligence. find
  20. Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. 2020b. OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings. arXiv. find
  21. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. find our card
  22. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 3484 10 find our card
  23. Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic Models for Analyzing and Detecting Biased Language. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. find our card
  24. Haoran Zhang, Amy X Lu, Mohamed Abdalla, Matthew McDermott, and Marzyeh Ghassemi. 2020. Hurtful words: Quantifying biases in clinical contextual word embeddings. In Proceedings of the ACM Conference on Health, Inference, and Learning. find
  25. Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019. Are Red Roses Red? Evaluating Consistency of Question-Answering Models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find
  26. Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, and Ahmed Awadallah. 2020. Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find
  27. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find our card
  28. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender Bias in Contextualized Word Embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find our card
  29. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBert, a distilled version of BERT: Smaller, faster, cheaper and lighter. In the Thirty-third Conference on Neural Information Processing Systems, 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing. find
  30. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find our card
  31. Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2019. The Woman Worked as a Babysitter: On Biases in Language Generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. find our card
  32. Gabriel Stanovsky, Noah A Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find our card
  33. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computati… find our card
  34. Yi Chern Tan and L Elisa Celis. 2019. Assessing Social and Intersectional Biases in Contextualized Word Representations. In Advances in Neural Information Processing Systems. find
  35. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP. find our card
  36. Thomas Wolf, L Debut, V Sanh, J Chaumond, C Delangue, A Moi, P Cistac, T Rault, R Louf, M Funtowicz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv. 3485 11 find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2020-unqovering-stereotyping-biases-underspecified-questions/