Elamur Research
☰
home / papers / 2019 / The Woman Worked as a Babysitter: On Biases in Language Generation

The Woman Worked as a Babysitter: On Biases in Language Generation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1339.pdf

Summary

Sheng et al. study how open-domain language generators portray different social groups. They found that off-the-shelf sentiment tools often miss socially loaded content: a sentence saying someone worked as a pimp is scored as neutral. So they propose regard as the bias metric. Regard is the social perception a sentence creates of a demographic. To measure it, they wrote 10 prefix templates in two bias contexts, respect and occupation. Each template was filled with one of six groups: man/woman, Black/White and gay/straight. They sampled GPT-2 continuations, and three annotators labeled 360 of them for both sentiment and regard. This left 302 usable examples, which they used to train a BERT-based regard classifier. They then scored 500 generations per group from GPT-2 and LM 1B. Both models treat groups unequally, and regard shows larger gaps in negative scores than sentiment does. This means sentiment-only audits can understate bias, which matters for every downstream system built on these models.

Key points

  • Models and tools: the models audited were GPT-2 (small) and Google's LM 1B, trained on the One Billion Word Benchmark. VADER was the main sentiment analyzer, and TextBlob was a baseline.
  • Annotation agreement: on the positive/neutral/negative classes, Fleiss' kappa was 0.60 for sentiment and 0.67 for regard. Spearman's correlation between annotators was 0.76 and 0.80. The dataset was split 212/60/30 for train/dev/test.
  • Sentiment vs. regard: VADER predictions match regard labels poorly in the occupation context (Spearman 0.54) but better in the respect context (0.69). The two human-annotated labels agree at 0.95 for respect but only 0.70 for occupation.
  • Classifier results: the BERT regard classifier reached about 0.8 accuracy. That is more than 20% above LSTM models and above a sentiment analyzer reused as a regard classifier.
  • Bias findings: with GPT-2, respect prompts led to more negative regard for Black, man and gay groups. Occupation prompts led to more negative regard for Black, woman and gay groups ("prostitute" came up often). LM 1B looked less biased overall. The annotations are public at github.com/ewsheng/nlg-bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

332 papers of our library cite it (of the 140673 whose bibliographies we read); 31 of them have a page here, 301 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1339 · arXiv 1909.01326

@inproceedings{sheng2019woman,
  title = {{The Woman Worked as a Babysitter: On Biases in Language Generation}},
  author = {Sheng, Emily and Chang, Kai-Wei and Natarajan, Premkumar and Peng, Nanyun},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1339},
  eprint = {1909.01326},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1339}
}

References

15 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, pages 4349–4357. find our card
  2. Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 123–129. ACM. find
  3. Clayton J Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Eighth international AAAI conference on weblogs and social media. find
  4. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence. find
  5. Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7378–7385. find
  6. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find our card
  7. Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang. 2019. Examining gender bias in languages with grammatical gender. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  8. Svetlana Kiritchenko and Saif M Mohammad. 2018. Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. In 7th Joint Conference on Lexical and Computational Semantics (SEM‘18). find our card
  9. Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press. find
  10. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Aman-charla, and Anupam Datta. 2018. Gender bias in neural natural language processing. arXiv preprint arXiv:1807.11714. find our card
  11. Saif Mohammad. 2016. A practical guide to sentiment annotation: Challenges and solutions. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 174–179. find
  12. Marcelo OR Prates, Pedro H Avelar, and Luı́s C Lamb. 2018. Assessing gender bias in machine translation: a case study with google translate. Neural Computing and Applications, pages 1–19. find
  13. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  14. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–1… find our card
  15. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building find