Elamur Research⌕
☰
home / papers / 2020 / AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings

AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings

Paper · 2020 · Anne Lauscher, Rafik Takieddin, Simone Paolo Ponzetto et al.

Summary

The paper asks whether Arabic word embeddings carry stereotypical associations, such as gender bias, as has been shown for English and several other languages. The authors build AraWEAT, an Arabic extension of the multilingual XWEAT framework. They translated the English WEAT term lists with Google Translate and had a native MSA speaker check and correct them, adding both masculine and feminine forms for words that are genderless in English. Four bias measures are applied: WEAT, ECT and BAT for explicit bias, and a K-Means++ clustering test for implicit bias. The embedding spaces vary along several axes: model (Skip-gram, CBOW, FastText), dimensionality, text genre (Wikipedia, news, Twitter), variety (Egyptian vs. MSA) and time (news from 2007 to 2017). The main findings are:

  • Egyptian Arabic vectors appear more biased than MSA vectors.
  • Implicit gender bias in news embeddings rises steadily over the decade.
  • The bias of a full corpus can be roughly predicted from the biases of its yearly slices.

This is presented as the first bias study of Arabic embedding spaces, and the test specifications are released publicly.

Key points

  • Tests kept: WEAT 1, 2, 7, 8 and 9. Tests 3–6 and 10 were dropped because they depend on personal names, which are hard to carry across languages. Test 9 was skipped for the pretrained spaces because under 20% of its vocabulary was found there.
  • Spaces examined:
    • pretrained 300-dim FastText (Arabic and Egyptian);
    • AraVec Skip-gram/CBOW trained on Wikipedia (MSA) and Twitter (mixed), in unigram and n-gram variants at 100 and 300 dimensions;
    • self-trained CBOW on the Arabic Leipzig News Corpora, using yearly slices of 300K and 1M sentences.
  • Embedding quality was also checked on SemEval 2017 Task 1 STS, with sentence vectors built by averaging word vectors.
  • In the diachronic news study:
    • The 300K-sentence slices showed no significant WEAT effect in any year.
    • The 1M-sentence slices showed significant gender bias (WEAT T7) in 2009, 2015 and 2017, with effect sizes of about .92–.97.
    • K-Means++ implicit bias grew across the whole period.
  • Bias scores of the whole news corpus correlated with the average of the yearly-slice scores at Pearson 0.66. User-generated text was not generally more biased, and dimensionality showed no clear trend except on T2.

Interesting ideas

  • I1. When translating a genderless English term into a language with grammatical gender, the authors include both gendered forms, so the translation step does not plant a bias of its own.
  • I2. An explicit association test is paired with an implicit test that checks whether two concept groups can be separated by clustering; together they catch bias that one probe alone would miss.
  • I3. A general quality score (STS) is reported next to every bias score, which shows that missing effects on small corpora may reflect weak vectors rather than an absence of bias.
  • I4. A large corpus is cut into disjoint time slices, each slice is measured, and the averaged slice scores track the whole-corpus score fairly well.
  • I5. A test is skipped when too little of its vocabulary is covered, instead of being reported on a sparse sample.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): when translating through the meta-world into gendered languages, unspecified gender in the graph should produce both variants or an explicitly neutral one, never a silent default.
  • I2 (learn): a "should not be separable" clustering check could serve as a negative control, flagging features that encode distinctions the model is not supposed to rely on.
  • I3 (learn): any diagnostic metric over silver data or a parser should be logged alongside a baseline quality score, so that low-quality runs are not misread as clean.
  • I4 (learn): evaluating silver data per source or period slice and aggregating could give cheap early estimates and expose drift before training on the full mix.
  • I5 (answer): school-test or fable QA evaluations could carry a lexicon-coverage threshold, so scores are reported only where the world/ontology covers enough of the needed terms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Anne Lauscher, Rafik Takieddin, Simone Paolo Ponzetto and Goran Glavaš. 2020. AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lauscher2020araweat,
  title = {{AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings}},
  author = {Anne Lauscher, Rafik Takieddin, Simone Paolo Ponzetto and Goran Glavaš},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.17/}
}

References

22 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Arthur and Sergei Vassilvitskii. 2007. K-means++: The advantages of careful seeding. In Proceedings of SODA, pages 1027–1035. ↗
  2. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of” bias” in nlp. In Proceedings of the 58th Meeting of the Association for Computational Linguistics, pages 5454–5476, Online, July. Association for Computational Linguistics.
  3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the ACL, 5:135–146. ↗
  4. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of NIPS, pages 4356– 4364. ↗
  5. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. ↗
  6. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, August.
  7. Sunipa Dev and Jeff Phillips. 2019. Attenuating bias in word vectors. In Proceedings of AISTATS. ↗
  8. Mohammed Elrazzaz, Shady Elbassuoni, Khaled Shaban, and Chadi Helwe. 2017. Methodical evaluation of Arabic word embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 454–458, Vancouver, Canada, July. Association for Computational Linguistics.
  9. Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019. Understanding undesirable word embedding associations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. ↗
  10. Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012. Building large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages. In In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC ’12).
  11. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of NAACL-HLT, pages 609–614. ↗
  12. Nizar Habash, Houda Bouamor, and Christine Chung. 2019. Automatic gender identification and reinflection in Arabic. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 155–165, Florence, Italy, August. Association for Computational Linguistics.
  13. Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and Simone Teufel. 2019. It’s all in the name: Mitigating gender bias with name-based counterfactual data substitution. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5270–5278. ↗
  14. Anne Lauscher and Goran Glavaš. 2019. Are we consistently biased? multidimensional analysis of biases in distributional word vectors. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (* SEM 2019), pages 85–91. find
  15. Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. 2020. A general framework for implicit and explicit debiasing of distributional word vector spaces. In Proceesings of AAAI, pages 8131–8138. ↗
  16. Katherine McCurdy and Oguz Serbetci. 2017. Grammatical gender associations outweigh topical gender bias in crosslinguistic word embeddings. In Proceedings of WiNLP. find
  17. Parkvall Mikael. 2007. Världens 100 största språk 2007 (the world’s 100 largest languages in 2007). National Encylkopedin. find
  18. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceesings of NIPS, pages 3111–3119. ↗
  19. Abu Bakr Mohammad, Kareem Eissa, and Samhaa El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265, 11. find
  20. Brian A. Nosek, Anthony G. Greenwald, and Mahzarin R. Banaji. 2002. Harvesting implicit group attitudes and beliefs from a demonstration web site. Group Dynamics, 6:101–115. ↗
  21. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532–1543.
  22. Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang. 2019. Examining gender bias in languages with grammatical gender. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5279–5287. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lauscher-2020-araweat-multidimensional-analysis-biases-arabic/