Elamur Research⌕
☰
home / papers / 2019 / Reducing Sentiment Bias in Language Models via Counterfactual Evaluation

Reducing Sentiment Bias in Language Models via Counterfactual Evaluation

Paper · 2019 · Po-Sen Huang, Huan Zhang, Ray Jiang et al.

Summary

The paper examines whether text produced by large language models carries sentiment bias, where the tone of a continuation shifts depending on which country, occupation or person name appears in the prompt. The authors use counterfactual evaluation. They swap the sensitive tokens in a conditioning context, generate continuations, score them with a sentiment classifier, and compare the resulting score distributions. Bias is measured with individual and group fairness metrics built on the Wasserstein-1 distance, which corresponds to demographic disparity averaged over all decision thresholds. Transformer models of GPT-2 scale, trained on WMT-19 news and WikiText-103, show substantial bias under these metrics. To reduce it, the authors add two regularizers on the model's latent representations: one based on embeddings and one based on sentiment prediction. Both lower bias while keeping perplexity and semantic similarity roughly unchanged. They also reveal a trade-off between fairness and relevance, and the sentiment-based variant gives the better balance. Human judgments agree well with the automatic metrics, which supports this evaluation approach for auditing generation systems.

Key points

  • The motivating example uses the 1.5B-parameter GPT-2. Prompts mentioning "baker" receive noticeably more positive continuations than prompts mentioning "accountant", as scored by the Google Cloud sentiment API.
  • The main experiments use 708M-parameter Transformer LMs trained on English WMT-19 news (40GB) and WikiText-103. The sensitive attributes tested are Country, Occupation and Name.
  • Sentiment is scored with three tools: the Google Cloud API, a BERT-based classifier and an opinion-word-based classifier. Their Spearman correlations with human ratings are 0.75, 0.79 and 0.67, and the semantic similarity metrics S.S. and S.S.c reach 0.72 and 0.63.
  • Under human evaluation, sentiment regularization cuts individual fairness (I.F.) scores from 0.333 to 0.056 on the designer/accountant prompts and from 0.291 to 0.155 on Libya/Iceland.
  • Perplexity stays close to the baseline on WMT-19 Occupation, at about 17.6–18.5 versus 17.9. At a similar semantic similarity level, sentiment regularization outperforms embedding regularization on I.F.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations🎓University College LondonLondon · United Kingdom107 works · 22,924 citations🎓University of California, Los AngelesLos Angeles · United States99 works · 56,333 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSentiment & emotion · 1,035 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama and Pushmeet Kohli. 2019. Reducing Sentiment Bias in Language Models via Counterfactual Evaluation. arXiv:1911.03064.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1911.03064 · arXiv 1911.03064

@misc{huang2019reducing,
  title = {{Reducing Sentiment Bias in Language Models via Counterfactual Evaluation}},
  author = {Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama and Pushmeet Kohli},
  year = {2019},
  doi = {10.48550/arxiv.1911.03064},
  eprint = {1911.03064},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1911.03064}
}

References

44 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019. Evaluating the underlying gender bias in contextualized word embeddings. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 33–39, Florence, Italy. Association for Computational Linguistics.
  2. A. Beutel, J. Chen, Z. Zhao, and E. H. Chi. 2017. Data decisions and theoretical implications when adversarially learning fair representations. CoRR, abs/1707.00075. — [arXiv:1707.00075](https://arxiv.org/abs/1707.00075) find
  3. Shikha Bordia and Samuel R. Bowman. 2019. Identifying and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 7–15, Minneapolis, Minnesota. Associati…
  4. T. Calders, F. Kamiran, and M. Pechenizkiy. 2009. Building classifiers with independency constraints. In International Conference on Data Mining Workshops, pages 13–18. find
  5. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  6. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
  7. S. Chiappa. 2019. Path-specific counterfactual fairness. In Thirty-Third AAAI Conference on Artificial Intelligence, pages 7801–7808. find
  8. S. Chiappa and William S. Isaac. 2019. A Causal Bayesian Networks Viewpoint on Fairness, volume 547 of IFIP AICT, pages 3–20. Springer Nature Switzerland. find
  9. S. Chiappa, R. Jiang, T. Stepleton, A. Pacchiano, H. Jiang, and J. Aslanides. 2020. A general approach to fairness with optimal transport. In Thirty-Fourth AAAI Conference on Artificial Intelligence. find
  10. Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa A. Weis, Kevin Swersky, Toniann Pitassi, and Richard S. Zemel. 2019. Flexibly fair representation learning by disentanglement. CoRR, abs/1906.02589. find
  11. Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  13. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In AIES, pages 67–73. ACM. find
  14. C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pages 214–226. find
  15. H. Edwards and A. Storkey. 2016. Censoring representations with an adversary. In 4th International Conference on Learning Representations. find
  16. Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H. Chi, and Alex Beutel. 2019. Counterfactual fairness in text classification through robustness. In AIES, pages 219–226. ACM. find
  17. M. Hardt, E. Price, and N. Srebro. 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems 29, pages 3315–3323.
  18. Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Anna Rohrbach. 2018. Women also snowboard: Overcoming bias in captioning models. In Proceedings of the European Conference on Computer Vision (ECCV), pages 771–787. find
  19. Dirk Hovy and Anders Søgaard. 2015. Tagging performance correlates with author age. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 483–488, Beijing, C…
  20. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 168–177. ACM. find
  21. Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. 2019. Wasserstein fair classification. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence. find
  22. Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. 2017. Avoiding discrimination through causal reasoning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural I… find
  23. Svetlana Kiritchenko and Saif Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43–53, New Orleans, Louisiana. Association for Computational Linguistics.
  24. M. J. Kusner, J. R. Loftus, C. Russell, and R. Silva. 2017. Counterfactual fairness. In Advances in Neural Information Processing Systems 30, pages 4069– 4079. find
  25. C. Louizos, K. Swersky, Y. Li, M. Welling, and R. Zemel. 2016. The variational fair autoencoder. In 4th International Conference on Learning Representations. find
  26. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2018. Gender bias in neural natural language processing. CoRR, abs/1807.11714.
  27. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.
  28. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
  29. Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun. 2019. Reducing gender bias in word-level language models with a gender-equalizing loss function. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 223–228, Florence, Italy. A… find
  30. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  31. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the Annual Meeting of the North American Association of Computational Linguistics (NAACL).
  32. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In EMNLP-IJCNLP, pages 3405–3410.
  33. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631–1642.
  34. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, and Jasmine Wang. 2019. Release strategies and the social impacts of language models. arXiv e-prints, page arXiv:1908.09203.
  35. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Association for Computational Linguistics.
  36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
  37. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. Empirical Methods in Natural Language Processing. find
  38. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems, pages 9051–9062.
  39. R. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork. 2013. Learning fair representations. In Proceedings of the 30th International Conference on Machine Learning, pages 325–333. find
  40. Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In AAAI/ACM Conference on AI, Ethics, and Society, pages 335–340. ACM. find
  41. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. DialoGPT: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536.
  42. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1…
  43. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2979–2989, Copenhagen, Denmark.…
  44. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2019-reducing-sentiment-bias-language-models/