Elamur Research⌕
☰
home / papers / 2026 / ConGA: Guidelines for Contextual Gender Annotation. a Framework for Annotating Gender in Machine Translation

ConGA: Guidelines for Contextual Gender Annotation. a Framework for Annotating Gender in Machine Translation

Summary

Translating from English, which rarely marks gender, into Italian, which requires gender agreement on nouns, articles, adjectives and participles, forces MT systems and LLMs to choose a gender that the source text often leaves open. These systems tend to fall back on masculine forms. The authors argue that bias research does not yet have a shared, linguistically grounded way to annotate and measure this. They propose ConGA (Contextual Gender Annotation), a set of word-level guidelines. On the English side, ConGA labels semantic gender as Masculine, Feminine or Ambiguous. On the Italian side, it records how grammatical gender is actually realised. Entity identifiers link mentions of the same referent across sentences. The scheme only covers animate (human) referents, because the gender of inanimate nouns is purely morphological and says nothing about social bias. The authors applied ConGA to the gENder-IT corpus and evaluated Italian translations from two models. The analysis shows systematic overuse of masculine forms and unreliable feminine output. The resulting gold standard and method support reproducible, cross-lingual evaluation of gender bias.

Key points

  • Tagset: English source tokens get M, F or A (ambiguous, where context does not justify assuming a gender). Italian target tokens get M or F. Entity-level IDs track each referent across sentences.
  • Data: the English side of gENder-IT (Vanmassenhove and Monti, 2021), an English–Italian challenge set built from MuST-SHE v1.04. Its earlier tags were removed and the data re-annotated to match the new guidelines.
  • Translations were generated with mBART and TowerInstruct-7B ("TowerLLM", thinkverse/towerinstruct), run locally through Ollama.
  • Main finding: both systems over-produce masculine forms and realise feminine gender inconsistently. The exact figures were not included in the excerpt we had.
  • The paper positions ConGA against WinoGender, WinoMT, MuST-SHE and English-only bias taxonomies (Hitti et al., 2019; Havens et al., 2022). ConGA moves from counting outputs toward describing where and how gender is encoded across languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5nai3xa6oon7

@inproceedings{rescigno2026conga,
  title = {{ConGA: Guidelines for Contextual Gender Annotation. a Framework for Annotating Gender in Machine Translation}},
  author = {Rescigno, Argentina Anna and Vanmassenhove, Eva and Monti, Johanna},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5nai3xa6oon7},
  url = {https://doi.org/10.63317/5nai3xa6oon7}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Luisa Bentivogli, Beatrice Savoldi, Matteo Negri, Mattia A. Di Gangi, Roldano Cattoni, and Marco Turchi. 2020. Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6923–693… find our card
  2. Geert Booij. 1996. Inherent versus contextual inflection and the split morphology hypothesis. In Geert Booij, Jaap Van Marle, Stephen Anderson, Mark Aronoff, Laurie Bauer, Mark Baker, Rudie Botha, Joan Bybee, Andrew Carstairs-McCarthy, Wolfgang Dressler, Jack Hoeksema, Rochelle Lieber, Peter Matthew… find
  3. Won Ik Cho, Ji Won Kim, Seok Min Kim, and Nam Soo Kim. 2019. On Measuring Gender Bias in Translation of Gender-neutral Pronouns. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 173–181, Florence, Italy. Association for Computational Linguistics. find
  4. Greville G. Corbett. 1991. Gender, 1 edition. Cambridge University Press. find
  5. Mostafa Elaraby, Ahmed Y. Tawfik, Mahmoud Khaled, Hany Hassan, and Aly Osama. 2018. Gender Aware Spoken Language Translation Applied to English-Arabic. ArXiv:1802.09287 [cs]. — [arXiv:1802.09287](https://arxiv.org/abs/1802.09287) find
  6. Joel Escudé Font and Marta R. Costa-jussà. 2019. Equalizing Gender Bias in Neural Machine Translation with Word Embeddings Techniques. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 147–154, Florence, Italy. Association for Computational Linguistics. find
  7. European Parliament. 2018. Gender-neutral language in the european parliament. Technical report, European Parliament, Luxembourg. Publications Office of the European Union. find
  8. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes. Proceedings of the National Academy of Sciences, 115(16). ArXiv:1711.08412 [cs]. — [arXiv:1711.08412](https://arxiv.org/abs/1711.08412) find
  9. Lucy Havens, Melissa Terras, Benjamin Bach, and Beatrice Alex. 2022. Uncertainty and Inclusivity in Gender Bias Annotation: An Annotation Taxonomy and Annotated Datasets of British English Text. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 30–57, S… find
  10. Yasmeen Hitti, Eunbee Jang, Ines Moreno, and Carolyne Pelletier. 2019. Proposed Taxonomy for Gender Bias in Text; A Filtering Methodology for the Gender Generalization Subtype. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 8–17, Florence, Italy. Associatio… find
  11. Daniel Kahneman. 2011. Thinking, Fast and Slow. Farrar, Straus and Giroux, New York. find
  12. Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pa… find our card
  13. Sally McConnell-Ginet. 2013. ‘ Gender and its relation to sex: The myth of ‘natural’ gender. In Greville G. Corbett, editor, The Expression of Gender, pages 3–38. DE GRUYTER. find
  14. Johanna Monti. 2017. Questioni di genere in traduzione automatica. In Al femminile: scritti linguistici in onore di Cristina Vallini, number 139 in Quaderni della rassegna. Franco Cesati Editore, Firenze. find
  15. Terttu Nevalainen and Helena Raumolin-Brunberg. 1993. Its strength and the beauty of it: The standardization of the third person neuter possessive in Early Modern English. In Dieter Stein and Ingrid Tieken-Boon van Ostade, editors, 1600 - 1800, pages 171–216. De Gruyter Mouton, Berlin, Boston. find
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  17. Marcelo O. R. Prates, Pedro H. Avelar, and Luís C. Lamb. 2020. Assessing Gender Bias in Machine Translation: a Case Study with Google Translate. Neural Computing and Applications, 32(10):6363–6381. find
  18. A. A. Rescigno, E. Vanmassenhove, J. Monti, and A. Way. 2020. A case study of natural gender phenomena in translation A comparison of google translate, bing microsoft translator and deepl for english to Italian, French and Spanish. In CEUR Workshop Proceedings, volume 2769. ISSN: 16130073. find
  19. Argentina Anna Rescigno and Johanna Monti. 2023. Gender Bias in Machine Translation: a statistical evaluation of Google Translate and DeepL for English, Italian and German. In Proceedings of the International Conference on Human-informed Translation and Interpreting Technology 2023, pages 1–11. INCO… find
  20. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender Bias in Coreference Resolution. ArXiv:1804.09301 [cs]. find our card
  21. Beatrice Savoldi, Jasmijn Bastings, Luisa Bentivogli, and Eva Vanmassenhove. 2025. A decade of gender bias in machine translation. Patterns, 6(6):101257. find
  22. Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. Gender Bias in Machine Translation. Transactions of the Association for Computational Linguistics, 9:845–874. Place: Cambridge, MA Publisher: MIT Press. find our card
  23. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating Gender Bias in Machine Translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics. find our card
  24. Eva Vanmassenhove. 2024. Gender Bias in Machine Translation and The Era of Large Language Models. ArXiv:2401.10016 [cs]. — [arXiv:2401.10016](https://arxiv.org/abs/2401.10016) find
  25. Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting Gender Right in Neural Machine Translation. In Proceedings of the 2018 find our card
  26. Conference on Empirical Methods in Natural Language Processing, pages 3003–3008, Brussels, Belgium. Association for Computational Linguistics. find
  27. Eva Vanmassenhove and Johanna Monti. 2021. gENder-IT: An Annotated English-Italian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomena. In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, pages 1–7, Online. Association for Computational Linguistics. find
  28. Eva Vanmassenhove, Dimitar Shterionov, and Andy Way. 2019. Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation. In Proceedings of Machine Translation Summit XVII: Research Track, pages 222–232, Dublin, Ireland. European Association for Machine Translation. find our card
  29. Tao Zhang, Ziqian Zeng, Yuxiang Xiao, Huiping Zhuang, Cen Chen, James Foulds, and Shimei Pan. 2024. GenderAlign: An Alignment Dataset for Mitigating Gender Bias in Large Language Models. ArXiv:2406.13925 [cs]. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rescigno-2026-conga-guidelines-contextual-gender-annotation/