Elamur Research⌕
☰
home / papers / 2022 / The Arabic Parallel Gender Corpus 2.0: Extensions and Analyses

The Arabic Parallel Gender Corpus 2.0: Extensions and Analyses

Summary

NLP systems such as machine translation usually produce a single output with no knowledge of who is speaking or who is being addressed. English "I" and "you" carry no gender, so a translation into Arabic, which marks gender heavily, has to pick one, often a stereotyped one (for example, a male doctor addressing a female nurse).

The authors extend APGC v1.0 (Habash et al., 2019), which covered only first-person singular speakers, into APGC v2.0. They drew English–Arabic pairs containing first- or second-person pronouns from OpenSubtitles 2018. Annotators labeled the speaker's and the addressee's grammatical gender at the sentence and word level, then rewrote gender-marked sentences into their counterparts. The result has four parallel Arabic versions (masculine/feminine "I" × masculine/feminine "you"), aligned with the English source and with English-to-Arabic MT output.

The corpus has more than 6.5 times as many sentences as v1.0 and exceeds 590K words. It is publicly released for gender identification, controlled generation and post-editing rewriting, and the authors also use it to examine gender bias in commercial MT.

Key points

  • Source pool: OpenSubtitles 2018 has 29.8M English–Arabic pairs. Filtering for English I/me/my/mine/myself and you/your/yours/yourself left 13.4M pairs: 21.1% with both persons, 42.5% first person only, 36.4% second person only.
  • Sentence labels pair a first-person value and a second-person value (e.g., FM, MB, BN), built from B (ambiguous), F, M, N and "!"-marked variants. Word-level tags are B, 1F, 1M, 2F, 2M and their marked variants.
  • The annotated original corpus has 58,035 sentences and 423,254 words. The balanced corpus comes in five versions (input plus targets MM, FM, MF, FF) with 80,326 sentences and 596,799 words.
  • Annotation guidelines set out several rules:
    • Annotators do not fix spelling, Hamza, Ta-Marbuta or punctuation, and do not add diacritics, so sentences that are ambiguous without diacritics are labeled B.
    • Proper nouns are treated as gender-ambiguous.
    • Badly broken sentences get a special label and are not rewritten.
    • Unsolvable cases are flagged HARD.
    • The English side is used only to understand the Arabic, never to decide gender.
  • An appendix lists the gender-specific English noun lemmas annotated in the data (e.g., waiter/waitress, actor/actress, nephew/niece).

Interesting ideas

  • I1. Gender is recorded separately for each discourse participant (speaker and addressee) as a combined label, rather than as one sentence-wide value, so the two persons can differ independently. — For Mova (world): Referents for "I" and "you" in the text world could each carry their own gender slot, filled only when the text supplies evidence (e.g., "she is a doctor").
  • I2. "Ambiguous" is a first-class value, and orthographic ambiguity is treated as useful because one string can serve users of either gender. — For Mova (back): When generating from the meaning graph, features the world leaves unresolved should stay unresolved, so the generator chooses neutral wording instead of inventing a gender.
  • I3. Each sentence is rewritten into minimal-pair variants that differ only in one participant's gender, which gives a controlled test bed for rewriting and bias measurement. — For Mova (learn): Build minimal-pair probes that flip only one participant's gender and check that only the dependent parts of the world and the answers change, as a negative control.
  • I4. Annotators get explicit escape hatches (a HARD flag, a label for broken input, a rule against correcting the source) instead of being forced to guess. — For Mova (learn): Witness parsers and LLM teacher tiers could output an explicit "undecidable" label that keeps such items out of silver training data instead of adding noisy guesses.
  • I5. A curated lexicon of inherently gendered English nouns (sir, widow, waitress, etc.) supplies the gender cues that English otherwise lacks. — For Mova (frames): Coreference could use such a lemma list as an explainable agreement constraint when linking he/she to nouns like "the waitress" or "the landlord".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

19 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 15 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alhafni2022arabic,
  title = {{The Arabic Parallel Gender Corpus 2.0: Extensions and Analyses}},
  author = {Alhafni, Bashar and Habash, Nizar and Bouamor, Houda},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/3pu77wdwndzt},
  url = {https://doi.org/10.63317/3pu77wdwndzt}
}

References

61 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alhafni, B., Habash, N., and Bouamor, H. (2020). Gender-aware reinflection using linguistically enhanced neural models. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 139–150, Barcelona, Spain (Online), December. Association for Computational Linguistics. find
  2. Alkuhlani, S. and Habash, N. (2011). A corpus for modeling morpho-syntactic agreement in Arabic: Gender, number and rationality. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 357–362, Portland, Oregon, USA, June. find our card
  3. Blodgett, S. L., Barocas, S., Daumé III, H., and Wallach, H. (2020). Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online, July. Association for Computational Linguistics. find our card
  4. Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V., and Kalai, A. T. (2016). Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In D. Lee, et al., editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc. find our card
  5. Bordia, S. and Bowman, S. R. (2019). Identifying and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 7–15, Minneapolis, Minnesota, June. find our card
  6. Cao, Y. T. and Daumé III, H. (2020). Toward gender-inclusive coreference resolution. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568–4595, Online, July. Association for Computational Linguistics. find our card
  7. Cercas Curry, A., Robertson, J., and Rieser, V. (2020). Conversational assistants and gender stereotypes: Public perceptions and desiderata for voice personas. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 72–78, Barcelona, Spain (Online), December. Association for Computational Linguistics. find
  8. Ciora, C., Iren, N., and Alikhani, M. (2021). Examining covert gender bias: A case study in Turkish and English machine translation models. In Proceedings of the 14th International Conference on Natural Language Generation, pages 55–63, Aberdeen, Scotland, UK, August. Association for Computational Linguistics. find
  9. Costa-jussà, M. R. and de Jorge, A. (2020). Fine-tuning neural machine translation on gender-balanced datasets. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 26–34, Barcelona, Spain (Online), December. Association for Computational Linguistics. find
  10. Dinan, E., Fan, A., Williams, A., Urbanek, J., Kiela, D., and Weston, J. (2020a). Queens are powerful too: Mitigating gender bias in dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8173–8188, Online, November. Association for Computational Linguistics. find our card
  11. Dinan, E., Fan, A., Wu, L., Weston, J., Kiela, D., and Williams, A. (2020b). Multi-dimensional gender bias classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 314–331, Online, November. Association for Computational Linguistics. find
  12. Elaraby, M., Tawfik, A. Y., Khaled, M., Hassan, H., and Osama, A. (2018). Gender aware spoken language translation applied to english-arabic. In 2018 2nd International Conference on Natural Language and Speech Processing (ICNLSP), pages 1–6. find
  13. Escudé Font, J. and Costa-jussà, M. R. (2019). Equalizing gender bias in neural machine translation with word embeddings techniques. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 147–154, Florence, Italy, August. Association for Computational Linguistics. find
  14. Gonen, H. and Goldberg, Y. (2019). Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. find our card
  15. Gonen, H. and Webster, K. (2020). Automatically identifying gender issues in machine translation using perturbations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1991–1995, Online, November. Association for Computational Linguistics. find
  16. Habash, N., Soudi, A., and Buckwalter, T. (2007). On Arabic Transliteration. In A. van den Bosch et al., editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods, pages 15–22. Springer, Netherlands. find
  17. Habash, N., Bouamor, H., and Chung, C. (2019). Automatic gender identification and reinflection in Arabic. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 155– 165, Florence, Italy, August. find
  18. Habash, N. Y. (2010). Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers. find
  19. Hall Maudslay, R., Gonen, H., Cotterell, R., and Teufel, S. (2019). It’s all in the name: Mitigating gender bias with name-based counterfactual data substitution. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5267–5275, Hong Kong, China, November. find
  20. Johnson, M. (2020). A scalable approach to reducing gender bias in google translate. Google AI Blog. find
  21. Katsarou, S., Rodríguez-Gálvez, B., and Shanahan, J. (2022). Measuring gender bias in contextualized embeddings. In Computer Sciences and Mathematics Forum, volume 3, page 3. MDPI. find
  22. Lauscher, A., Takieddin, R., Ponzetto, S. P., and Glavaš, G. (2020). AraWEAT: Multidimensional analysis of biases in Arabic word embeddings. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 192–199, Barcelona, Spain (Online), December. Association for Computational Linguistics. find
  23. Lison, P. and Tiedemann, J. (2016). OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923–929, Portorož, Slovenia, May. European Language Resources Association (ELRA). find our card
  24. Liu, H., Dacon, J., Fan, W., Liu, H., Liu, Z., and Tang, J. (2020a). Does gender matter? towards fairness in dialogue systems. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4403–4416, Barcelona, Spain (Online), December. International Committee on Computational Linguistics. find
  25. Liu, H., Wang, W., Wang, Y., Liu, H., Liu, Z., and Tang, J. (2020b). Mitigating gender bias for neural dialogue generation with adversarial learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 893–903, Online, November. Association for Computational Linguistics. find
  26. Lu, K., Mardziel, P., Wu, F., Amancharla, P., and Datta, A. (2018). Gender bias in neural natural language processing. find our card
  27. Maass, A. and Arcuri, L. (1996). Language and stereotyping. Stereotypes and stereotyping, pages 193– 226. find
  28. Manzini, T., Yao Chong, L., Black, A. W., and Tsvetkov, Y. (2019). Black is to criminal as caucasian is to police: Detecting and removing multi-class bias in word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 615–621, Minneapolis, Minnesota, June. Association for Computational Linguistics. find
  29. Mehrabi, N., Gowda, T., Morstatter, F., Peng, N., and Galstyan, A. (2019). Man is to person as woman is to location: Measuring gender bias in named entity recognition. find
  30. Menegatti, M. and Rubini, M. (2017). Gender bias and sexism in language. In Oxford Research Encyclopedia of Communication. Oxford University Press. find
  31. Moryossef, A., Aharoni, R., and Goldberg, Y. (2019). Filling gender & number gaps in neural machine translation with black-box context injection. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 49–54, Florence, Italy, August. find
  32. Obeid, O., Zalmout, N., Khalifa, S., Taji, D., Oudah, M., Alhafni, B., Inoue, G., Eryani, F., Erdmann, A., and Habash, N. (2020). CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 7022– 7032, Marseille, France, May. European Language Resources Association. find our card
  33. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the Conference of the Association for Computational Linguistics (ACL), pages 311–318, Philadelphia, Pennsylvania, USA. find our card
  34. Pasha, A., Al-Badrashiny, M., Diab, M., Kholy, A. E., Eskander, R., Habash, N., Pooleery, M., Rambow, O., and Roth, R. (2014). Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), pages 1094–1101, Reykjavik, Iceland. find our card
  35. Post, M. (2018). A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium, October. find our card
  36. Rabinovich, E., Patel, R. N., Mirkin, S., Specia, L., and Wintner, S. (2017). Personalized machine translation: Preserving original author traits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1074–1084, Valencia, Spain, April. find
  37. Rudinger, R., Naradowsky, J., Leonard, B., and Van Durme, B. (2018). Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–14, New Orleans, Louisiana, June. find our card
  38. Saunders, D. and Byrne, B. (2020). Reducing gender bias in neural machine translation as a domain adaptation problem. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7724–7736, Online, July. Association for Computational Linguistics. find our card
  39. Saunders, D., Sallis, R., and Byrne, B. (2020). Neural machine translation doesn’t translate gender coreference right unless you make it. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 35–43, Barcelona, Spain (Online), December. Association for Computational Linguistics. find our card
  40. Saunders, D., Sallis, R., and Byrne, B. (2021). First the worst: Finding better gender translations during beam search. find
  41. Savoldi, B., Gaido, M., Bentivogli, L., Negri, M., and Turchi, M. (2021). Gender Bias in Machine Translation. Transactions of the Association for Computational Linguistics, 9:845–874, 08. find our card
  42. Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. (2019). The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3407–3412, Hong Kong, China, November. find our card
  43. Sheng, E., Arnold, J., Yu, Z., Chang, K.-W., and Peng, N. (2021a). Revealing persona biases in dialogue systems. find
  44. Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. (2021b). "nice try, kiddo": Investigating ad hominems in dialogue responses. find
  45. SpaCy. (2017). spacy - industrial-strength natural language processing in python. find
  46. Stafanovičs, A., Bergmanis, T., and Pinnis, M. (2020). Mitigating gender bias in machine translation with target gender annotations. find our card
  47. Stanovsky, G., Smith, N. A., and Zettlemoyer, L. (2019). Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy, July. find our card
  48. Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.-W., and Wang, W. Y. (2019). Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1630– 1640, Florence, Italy, July. Association for Computational Linguistics. find our card
  49. Sun, T., Webster, K., Shah, A., Wang, W. Y., and Johnson, M. (2021). They, them, theirs: Rewriting with gender-neutral english. find
  50. Taji, D., Khalifa, S., Obeid, O., Eryani, F., and Habash, N. (2018). An Arabic Morphological Analyzer and Generator with Copious Features. In Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology (SIGMORPHON), pages 140–150. find our card
  51. Vanmassenhove, E. and Monti, J. (2021). gENder-IT: An annotated English-Italian parallel challenge set for cross-linguistic natural gender phenomena. In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pages 1–7, Online, August. Association for Computational Linguistics. find
  52. Vanmassenhove, E., Hardmeier, C., and Way, A. (2018). Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3003–3008, Brussels, Belgium, October-November. find our card
  53. Vanmassenhove, E., Emmery, C., and Shterionov, D. (2021). NeuTral Rewriter: A rule-based and neural approach to automatic rewriting into gender neutral alternatives. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8940–8948, Online and Punta Cana, Dominican Republic, November. Association for Computational Linguistics. find
  54. Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. (2020). Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, et al., editors, Advances in Neural Information Processing Systems, volume 33, pages 12388–12401. Curran Associates, Inc. find
  55. Zaghouani, W., Mohit, B., Habash, N., Obeid, O., Tomeh, N., Rozovskaya, A., Farra, N., Alkuhlani, S., and Oflazer, K. (2014). Large Scale Arabic Error Annotation: Guidelines and Framework. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland. find our card
  56. Zhao, J., Wang, T., Yatskar, M., Ordonez, V., and Chang, K.-W. (2018a). Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana, June. find our card
  57. Zhao, J., Zhou, Y., Li, Z., Wang, W., and Chang, K.-W. (2018b). Learning gender-neutral word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4847–4853, Brussels, Belgium, October-November. find our card
  58. Zhao, J., Mukherjee, S., Hosseini, S., Chang, K.-W., and Hassan Awadallah, A. (2020). Gender bias in multilingual embeddings and cross-lingual transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2896–2907, Online, July. Association for Computational Linguistics. find
  59. Zmigrod, R., Mielke, S. J., Wallach, H., and Cotterell, R. (2019). Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1651–1661, Florence, Italy, July. 9. find our card
  60. Language Resource References Habash, Nizar and Bouamor, Houda and Chung, Christine. (2019). Automatic Gender Identification and Reinflection in Arabic. find
  61. Lison, Pierre and Tiedemann, Jörg. (2016). OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. European Language Resources Association (ELRA). find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alhafni-2022-arabic-parallel-gender-corpus-2/