Elamur Research⌕
☰
home / papers / 2022 / French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English

French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English

Summary

Research on social bias in NLP has mostly focused on English and the social context of the United States. This paper builds a French counterpart to CrowS-pairs, the US-centred benchmark of minimal sentence pairs. Each pair contrasts a stereotype about a disadvantaged group with the same statement about an advantaged group, and a masked language model is checked for a consistent preference for the stereotyped version.

The authors, native French speakers, translated the original pairs. They adapted culturally loaded elements such as names, places and food, and flagged items that could not be transferred. Additional French-specific stereotypes came from volunteers on the LanguageARC citizen-science platform. The authors also repaired structural flaws they found in the English pairs, which produced a revised English version.

Tests on three French models and one multilingual masked LM show that they prefer stereotyped sentences in most bias categories. The work offers a reusable recipe for extending bias benchmarks to new languages and cultures, along with a critical audit of the original dataset.

Key points

  • The French set has 1,677 sentence pairs. Of these, 1,467 are translated from the 1,508 original CrowS-pairs and 210 were newly crowdsourced and then translated into English.
  • All nine bias types from CrowS-pairs are represented (disability, age, gender, religion, nationality, physical appearance, sexual orientation, ethnicity/color, and socioeconomic status/occupation), and an additional "other" category is included.
  • Translation and adaptation touched 670 pairs in total, with first-name substitution the most frequent change. Some items were marked untranslatable because they were US-specific, such as references to the KKK.
  • LanguageARC collection ran from August to October 2021 and gathered:
    • 229 raw stereotypes from 26 users,
    • 426 fluency judgments from 13 users,
    • 2,599 bias-category judgments from 52 users.
  • The data and code are released on GitLab under CC BY-SA 4.0. The release includes the French pairs, their English translations and the corrected English CrowS-pairs.

Interesting ideas

  • I1.

    Adapt culture, not just words. The translators swapped names, foods, countries and neighbourhoods for local equivalents that carry the same social connotation. For example, a US housing reference became the Paris suburb "neuf-trois." Any port of a bias or commonsense benchmark, including into Ukrainian, needs this kind of adaptation to stay valid.

  • I2.

    A typology of minimal-pair defects. The authors name three recurring flaws:

    • non-minimal pairs, where extra words differ between the sentences;
    • double switches, where a second change such as a negation alters the meaning;
    • bias mismatches, where the two halves of a pair belong to different bias categories.

    This works as a ready-made checklist for auditing any contrastive evaluation set.

  • I3.

    Feed fixes back into the source. Problems spotted during translation were used to correct the original English data. Translation thus doubles as a quality review of the source benchmark.

  • I4.

    Split crowdsourcing into small tasks. The volunteer platform separated three jobs: writing new stereotypes, rating translation fluency and assigning bias categories. This makes participation easy and adds a validation layer. A caveat is that the prompts produced almost no anti-stereotype sentences, so task wording deserves care.

  • I5.

    Count untranslatable items as a result. Logging which items cannot cross cultures turns the translation effort into evidence of how US-centric the original benchmark is. That measurement is informative in its own right for comparing biases across languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cited by

69 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 67 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.583

@inproceedings{neveol2022french,
  title = {{French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English}},
  author = {Névéol, Aurélie and Dupont, Yoann and Bezançon, Julien and Fort, Karën},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.583},
  url = {https://doi.org/10.18653/v1/2022.acl-long.583}
}

References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marion Bartl, Malvina Nissim, and Albert Gatt. 2020. Unmasking contextual stereotypes: Measuring and mitigating BERT’s gender bias. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 1–16, Barcelona, Spain (Online). Association for Computational Linguistics. find
  2. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? . In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Associ… find
  3. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics. find our card
  4. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computa… find
  5. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  6. Oralie Cattan, Christophe Servan, and Sophie Rosset. 2021. On the usability of transformers-based models for a french question-answering task. In Recent Advances in Natural Language Processing (RANLP). find
  7. Jon Chamberlain, Karën Fort, Udo Kruschwitz, Mathieu Lafourcade, and Massimo Poesio. 2013. Using games to create language resources: Successes and limitations of the approach. In Iryna Gurevych and Jungi Kim, editors, The People’s Web Meets NLP, Theory and Applications of Natural Language Processing… find
  8. K. Bretonnel Cohen, Jingbo Xia, Pierre Zweigenbaum, Tiffany Callahan, Orin Hargraves, Foster Goss, Nancy Ide, Aurélie Névéol, Cyril Grouin, and Lawrence E. Hunter. 2018. Three Dimensions of Reproducibility in Natural Language Processing. In Proceedings of LREC, page 156–165. find
  9. Hal Daumé III. 2016. Language bias and black sheep. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  11. James Fiumara, Christopher Cieri, Jonathan Wright, and Mark Liberman. 2020. LanguageARC: Developing language resources through citizen linguistics. In Proceedings of the LREC 2020 Workshop on “Citizen Linguistics in Language Resource Development”, pages 1–6, Marseille, France. European Language Reso… find
  12. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for… find our card
  13. Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sanchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metrics do not correlate with application bias. In Proceedings of ACL 2021. find our card
  14. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Cardiff, United Kingdom. Leibniz-Institut für Deutsche Sprach… find
  15. Dirk Hovy and Shannon L. Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598, Berlin, Germany. Association for Computational Linguistics. find our card
  16. Ann Irvine, John Morgan, Marine Carpuat, Hal Daumé III, and Dragos Munteanu. 2013. Measuring machine translation errors in new domains. Transactions of the Association for Computational Linguistics, 1:429–440. find our card
  17. Mascha Kurpicz-Briki. 2020. Cultural differences in bias? origin and gender bias in pre-trained german and french word embeddings. In Proceedings of the 5th Swiss Text Analytics Conference (SwissText) & 16th Conference on Natural Language Processing (KONVENS), volume 2624, Zurich, Switzerland (held… find
  18. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. find
  19. Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020. FlauBERT: Unsupervised language model pre-training for French. In Proceedings of the 12th Language Resources and Evaluation Conference,… find our card
  20. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2021. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computationa… find our card
  21. Jean-Paul Vinay and Jean Darbelnet. 1958. Stylistique comparée du français et de l’anglais [Texte imprimé] : méthode de traduction / J.P. Vinay, J. Darbelnet. Bibliothèque de stylistique comparée. Didier, Paris. find
  22. Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, and Ahmed Hassan Awadallah. 2020. Gender bias in multilingual embeddings and cross-lingual transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2896–2907, Online. Association for… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/neveol-2022-french-crows-pairs-extending-challenge/