Elamur Research⌕
☰
home / papers / 2025 / From Measurement to Mitigation: Exploring the Transferability of Debiasing Approaches to Gender Bias in Maltese Language Models

From Measurement to Mitigation: Exploring the Transferability of Debiasing Approaches to Gender Bias in Maltese Language Models

Paper · 2025 · Melanie Galea and Claudia Borg

Summary

Most work on detecting and reducing gender bias in language models targets English. This paper asks whether those English methods still work for Maltese, a low-resource language with a Semitic core, Romance influences and pervasive grammatical gender. The authors examine two public Maltese BERT-style models: the monolingual BERTu and the multilingual mBERTu. They measure bias three ways: a Maltese-adapted CrowS-Pairs, a Maltese translation of selected SEAT tests, and masked-slot templates built from earlier probing studies. Next, they test four bias-reduction techniques, namely GuiDebias, Auto-Debias, Dropout Regularization and Counterfactual Data Augmentation. CDA reduced bias most consistently. Dropout helped more on the multilingual model. Auto-Debias worked on the monolingual model but made the multilingual one worse. GuiDebias increased bias in both. The metrics often disagreed, which supports judging debiasing with several measures. The work matters because it shows that English-centric fairness methods carry over unevenly to morphologically rich languages. It also releases code and Maltese evaluation and debiasing data that others can reuse.

Key points

  • Models studied: BERTu and mBERTu (Micallef et al., 2022). English BERT and mBERT served as comparison points on the English CrowS-Pairs.
  • Measurement used three instruments:
    • the extended multilingual CrowS-Pairs (Fort et al., 2024), which has nine bias categories;
    • SEAT6a/6b/7a/7b/8a/8b, machine-translated and then corrected by native speakers;
    • masked-slot templates adapted from BEC-Pro, DisCo and a Dutch RoBERTa study, with name pairs such as John/Ġanni and Jane/Ġovanna.
  • CDA used unseen sentences from FLORES+ and a subset of Korpus Malti v4.2. It was the best method overall, but its word-list swaps left gender agreement broken in many sentences.
  • For Dropout Regularization, the best settings for both models were h = 0.2 and a = 0.15. GuiDebias did not transfer, and t-SNE plots showed gender separation surviving in BERTu even after CDA.
  • All data, including the Maltese Debiasing Dataset, is released at github.com/MLRS/Malti-Bias.

Interesting ideas

  • I1. When a masked model filled template slots with Maltese object-pronoun suffixes instead of nouns, the authors added the article "il-" before the slot to force a noun reading. This shows that a small grammatical cue can control which word class a probe returns.
  • I2. CDA's weakness is that it flips only the listed gendered words and leaves agreeing verbs and other words unchanged. Counterfactual edits therefore need to propagate agreement, not just substitute strings.
  • I3. The study kept finding that CrowS-Pairs, SEAT and t-SNE pointed in different directions. A single bias or quality metric can mislead, so conclusions should rest on several independent probes.
  • I4. Small benchmarks were localised cheaply: machine translation first, then native-speaker post-editing to fix grammar and structure. This makes a bias test usable in a new language at low cost.
  • I5. Pairing English names with their local equivalents (John/Ġanni) separates the effect of the name itself from the effect of the template.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When masked models act as witnesses or teachers on fill-in queries, insert a part-of-speech-forcing cue (e.g., a determiner) so their answers are comparable and do not drift into the wrong category.
  • I2 (back): Make the gender (or number) flip on the meaning graph and then regenerate the text, so agreement is produced consistently and the result is a clean counterfactual pair.
  • I3 (learn): Score each training or repair change against multiple unrelated test sets plus negative controls, and flag changes where the metrics disagree instead of trusting one number.
  • I4 (answer): Build or extend small school-style test sets the same way, generating drafts automatically and hand-correcting them, rather than writing every item from scratch.
  • I5 (frames): In coreference tests, swap names across cultures and genders while keeping the text fixed, to check whether pronoun resolution leans on name stereotypes rather than syntax and discourse.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MaltaMsida · Malta32 works · 895 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSemitic languages · 3,066 worksMultilingual & cross-lingual NLP · 882 worksBias, fairness & safety · 587 works

Cite

Cite as Melanie Galea and Claudia Borg. 2025. From Measurement to Mitigation: Exploring the Transferability of Debiasing Approaches to Gender Bias in Maltese Language Models. Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing (GeBNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.gebnlp-1.26

@inproceedings{galea2025measurement,
  title = {{From Measurement to Mitigation: Exploring the Transferability of Debiasing Approaches to Gender Bias in Maltese Language Models}},
  author = {Melanie Galea and Claudia Borg},
  booktitle = {Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing (GeBNLP)},
  year = {2025},
  doi = {10.18653/v1/2025.gebnlp-1.26},
  url = {https://doi.org/10.18653/v1/2025.gebnlp-1.26}
}

References

24 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marion Bartl, Malvina Nissim, and Albert Gatt. 2020. Unmasking contextual stereotypes: Measuring and mitigating BERT’s gender bias. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 1–16, Barcelona, Spain (Online). Association for Computational Linguistics.
  2. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery. ↗
  3. Su Lin Blodgett and Brendan O’Connor. 2017. Racial disparity in natural language processing: A case study of social media african-american english. CoRR, abs/1707.00061. ↗
  4. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
  5. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. CoRR, abs/2108.07258. ↗
  6. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. ↗
  7. Rodrigo Alejandro Chávez Mulsa and Gerasimos Spanakis. 2020. Evaluating bias in Dutch word embeddings. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 56–71, Barcelona, Spain (Online). Association for Computational Linguistics. find
  8. Pieter Delobelle, Ewoenam Tokpo, Toon Calders, and Bettina Berendt. 2022. Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1693–1706, Seattle, United States. Association for Computational Linguistics. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  10. Karen Fort, Laura Alonso Alemany, Luciana Benotti, Julien Bezançon, Claudia Borg, Marthese Borg, Yongjian Chen, Fanny Ducel, Yoann Dupont, Guido Ivetta, et al. 2024. Your stereotypical mileage may vary: Practical challenges of evaluating biases in multiple languages and cultural contexts. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 17764– 17769, Torino, Italia. ELRA and ICCL. find
  11. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097– 1179.
  12. Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, page 122–133, New York, NY, USA. Association for Computing Machinery. find
  13. Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In Proceedings of The ACM Collective Intelligence Conference, CI ’23, page 12–24, New York, NY, USA. Association for Computing Machinery. find
  14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. ↗
  15. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2018. Gender bias in neural natural language processing. CoRR, abs/1807.11714. ↗
  16. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019a. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622–628, Minneapolis, Minnesota. Association for Computational Linguistics.
  17. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019b. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622–628, Minneapolis, Minnesota. Association for Computational Linguistics.
  18. Nicholas Meade, Spandana Gella, Devamanyu Hazarika, Prakhar Gupta, Di Jin, Siva Reddy, Yang Liu, and Dilek Hakkani-Tur. 2023. Using in-context learning to improve dialogue safety. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11882–11910, Singapore. Association for Computational Linguistics. find
  19. Kurt Micallef, Albert Gatt, Marc Tanti, Lonneke van der Plas, and Claudia Borg. 2022. Pre-training data quality and quantity for a low-resource language: New corpus and BERT models for Maltese. In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pages 90–101, Hybrid. Association for Computational Linguistics.
  20. Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. 2020. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint arXiv:2010.06032. ↗
  21. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903. ↗
  22. Tae-Jin Woo, Woo-Jeoung Nam, Yeong-Joon Ju, and Seong-Whan Lee. 2023. Compensatory debiasing for gender imbalances in language models. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. find
  23. Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, page 335–340, New York, NY, USA. Association for Computing Machinery. find
  24. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. CoRR, abs/1804.06876. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/galea-2025-measurement-mitigation-exploring-transferability-debiasing/