Elamur Research⌕
☰
home / papers / 2026 / Quality and Appropriateness of Large Text Datasets for Irish NLP

Quality and Appropriateness of Large Text Datasets for Irish NLP

Summary

Irish is an official EU language but remains under-resourced. Most Irish text available for training NLP models comes from large, automatically assembled web corpora, and their quality has rarely been checked. Authors from the ADAPT Centre (Dublin City University) audit 14 such resources. These are the Irish portions of four monolingual or multilingual collections from HuggingFace and ten English–Irish parallel corpora from OPUS. Curated, linguistically annotated and speech corpora, which are presumed to be of higher quality, were deliberately excluded. Five annotators with at least B2-level Irish judged random samples on three dimensions: appropriateness (using the six-label scheme of Kreutzer et al., 2022), harmfulness, and non-standardness. Automatic metrics and LLM-as-judge scoring complemented the manual review. The authors also compared the preprocessing steps each corpus creator reported. The annotated examples reveal wrong or misaligned translations, boilerplate, incoherent machine-like Irish, pornographic spam and personal data. The provided excerpt omits the quantitative results. The audit matters because multilingual LLMs that support Irish are probably trained on exactly these corpora.

Key points

  • Corpus sizes (Irish side, millions of tokens): NLLB 250, CulturaX 216, C4 103, Paracrawl 63, HPLT 59, CCMatrix 21, OSCAR 9, Wikipedia 8, and Tatoeba only 0.01.
  • Reported preprocessing: Language detection was the most frequent step (11 of 14 datasets), and most other steps involve removing text. Spelling or OCR correction and transliteration were rare.
  • Annotation guidelines: These were refined over three rounds. Harmfulness used five categories, following Chehbouni et al. (2025a). Non-standardness labels covered deviation from the Caighdeán Oifigiúil, code-switching, user-generated content and non-text elements. The authors stress that non-standardness is not treated as poor quality.
  • Sample volume: About 150,710 Irish tokens were reviewed in total (14,933 parallel and 135,777 monolingual). Parallel pairs averaged about 15 Irish words, while monolingual documents averaged 339 words and were not length-normalised.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3sxe9j64u492

@inproceedings{walsh2026quality,
  title = {{Quality and Appropriateness of Large Text Datasets for Irish NLP}},
  author = {Walsh, Abigail and Andrade, Mark and Adkins, Jane Lauren and O'Connell, Ornait and O'Connor, Éanna and Rushe, Ellen and Davis, Brian},
  booktitle = {Proceedings of the SIGUL 2026 Joint Workshop with ELE, EURALI, and DCLRL: Towards Inclusivity and Equality: Language Resources and Technologies for Under-Resourced and Endangered Languages},
  year = {2026},
  doi = {10.63317/3sxe9j64u492},
  url = {https://doi.org/10.63317/3sxe9j64u492}
}

References

66 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jane Adkins, Hugo Collins, Joachim Wagner, Abigail Walsh, and Brian Davis. 2025. Named Entity Recognition for the Irish Language. In Proceedings of the 21st Workshop on Multiword Expressions (MWE 2025), pages 82–96, Albuquerque, New Mexico, U.S.A. Association for Computational Linguistics. find
  2. Giulio Angiani, Laura Ferrari, Tomaso Fontanini, Paolo Fornacciari, Eleonora Iotti, Federico Magliani, and Stefano Manicardi. 2016. A Comparison between Preprocessing Techniques for Sentiment Analysis in Twitter. In International Workshop on Knowledge Discovery on the Web. find
  3. James Barry, Joachim Wagner, Lauren Cassidy, Alan Cowap, Teresa Lynn, Abigail Walsh, Mícheál J. Ó Meachair, and Jennifer Foster. 2022. gaBERT — an Irish Language Model. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4774–4788, Marseille, France. European Languag… find
  4. Joachim Baumann, Paul Röttger, Aleksandra Urman, Albert Wendsjö, Flor Miriam Plaza del Arco, Johannes B. Gruber, and Dirk Hovy. 2025. Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation. find
  5. Abeba Birhane, Vinay Prabhu, Sang Han, and Vishnu Naresh Boddeti. 2023. On Hate Scaling Laws for Data-swamps. arXiv preprint arXiv:2306.13141. find
  6. Jose Camacho-Collados and Mohammad Taher Pilehvar. 2018. On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis. find
  7. Lauren Cassidy, Teresa Lynn, James Barry, and Jennifer Foster. 2022. TwittIrish: A Universal Dependencies Treebank of Tweets in Modern Irish. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6869–6884, Dublin, Ireland. Associat… find
  8. Khaoula Chehbouni, Jonathan Colaço Carr, Yash More, Jackie CK Cheung, and Golnoosh Farnadi. 2025a. Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human… find
  9. Khaoula Chehbouni, Mohammed Haddou, Jackie Chi Kit Cheung, and Golnoosh Farnadi. 2025b. Neither Valid nor Reliable? Investigating the Use of LLMs as Judges. arXiv preprint arXiv:2508.18076. — [arXiv:2508.18076](https://arxiv.org/abs/2508.18076) find
  10. Teresa Clifford, Abigail Walsh, Brian Davis, and Mícheál J. Ó Meachair. 2025. Gaeilge Bhriste ó Shamhlacha Cliste: How Clever Are LLMs When Translating Irish Text? In Proceedings of the 5th Celtic Language Technology Workshop, pages 46–51, Abu Dhabi [Virtual Workshop]. International Committee on Com… find
  11. Elliott Lash Fangzhe Qiu Nora White Siobhán Barrett Aaron Griffith Romanas Bulatovas Francesco Felici Ellen Ganly Truc Ha Nguyen Lars Nooij David Stifter, Bernhard Bauer. 2021. Corpus palaeohibernicum (corph) v1.0. http://chronhib.maynoothuniversity.ie. ↗
  12. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. find
  13. Meghan Dowling, Teresa Lynn, and Andy Way. 2019. Leveraging Backtranslation to Improve Machine Translation for Gaelic Languages. In Proceedings of the Celtic Language Technology Workshop, pages 58–62, Dublin, Ireland. European Association for Machine Translation. find
  14. Fiona Farr, Brona Murphy, and Anne O’Keeffe. 2004. The Limerick Corpus of Irish English: Design, Description and Application. Teanga: The Irish Yearbook of Applied Linguistics, 21:5–29. find
  15. Christoph Gröger. 2021. There is no AI Without Data. Communications of the ACM, 11:98–108. find
  16. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhou Wang, Kun Zhang, Zhouchi Lin, Bowen Zhang, Lionel Ni, Wen Gao, Yuanzhuo Wang, and Jian Guo. 2026. A Survey on LLM-as-a-Judge. The Innovation. find
  17. HuggingFace. 2025. Huggingface. find
  18. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp… find our card
  19. John Judge, Ailbhe Ní Chasaide, Elaine Uí Dhonnchadha, Rose Ní Dhubhda, and Kevin P. Scannell. 2012. The Irish Language in the Digital Age. find
  20. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. find
  21. Amr Keleg, Matthias Lindemann, Danyang Liu, Wanqiu Long, and Bonnie L. Webber. 2022. Automatically Discarding Straplines to Improve Data Quality for Abstractive News Summarization. In Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP, pages 42–51, Dublin, Ireland. Associa… find
  22. Adam Kilgarriff, Michael Rundell, and Elaine Uí Dhonnchadha. 2006. Efficient Corpus Development for Lexicography: Building the New Corpus for Ireland. Language Resources Evaluation, 40:127–152. find
  23. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro… find our card
  24. Klaus Krippendorff. 2011. Computing Krippendorff’s Alpha-Reliability. find
  25. Séamus Lankford, Haithem Afli, Órla Ní Loinsigh, and Andy Way. 2022. gaHealth: An English–Irish Bilingual Corpus of Health Data. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6753–6758, Marseille, France. European Language Resources Association. find
  26. Seamus Lankford, Haithem Afli, and Andy Way. 2021. Machine Translation in the Covid Domain: an English-Irish Case Study for LoResMT 2021. In Proceedings of the 4th Workshop on Technologies for MT of Low Resource Languages (LoResMT2021), pages 144–150, Virtual. Association for Machine Translation in… find
  27. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods. arXiv preprint arXiv:2412.05579. find our card
  28. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks Are All You Need II: Phi-1.5 Technical Report. find our card
  29. Liam Lonergan, Mengjie Qian, Harald Berthelsen, Andy Murphy, Christoph Wendler, Neasa Ní Chiaráin, Christer Gobl, and Ailbhe Ní Chasaide. 2022. Automatic Speech Recognition for Irish: the ABAIR-ÉIST System. In Proceedings of the 4th Celtic Language Technology Workshop within LREC2022, pages 47–51, M… find
  30. Teresa Lynn. 2022. Report on the Irish Language. https://european-language-equality.eu/deliverables/. Technical Report D1.20, European Language Equality Project. ↗
  31. Meta. 2024. meta-llama/llama-3.1-8b-instruct. find
  32. Microsoft. 2024. microsoft/phi‑4. find
  33. MigoXLab. 2024. Dingo: A Comprehensive AI Data Quality Evaluation Tool for Large Models. https://github.com/MigoXLab/dingo. ↗
  34. Mistral AI. 2024. mistralai/mistral-7b-instruct-v0.2. find
  35. Yida Mu, Mali Jin, Xingyi Song, and Nikolaos Aletras. 2024. Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Research. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12477–12492, Miami, Florida… find
  36. Gearóid Ó Cleircín, Anna Bale, and Brian Ó Raghallaigh. 2014. Dúchas.ie: Ré Nua i Stair Chnu-asach Bhéaloideas Éireann. find
  37. Mícheál Ó Meachair, Úna Bhreathnach, and Gearóid Ó Cleircín. 2022. Introducing the National Corpus of Irish Project. In Proceedings of the 4th Celtic Language Technology Workshop within LREC2022, pages 99–103, Marseille, France. European Language Resources Association. find
  38. Akintunde Oladipo, Mofetoluwa Adeyemi, Orevaoghene Ahia, Abraham Toluwalase Owodunni, Odunayo Ogundepo, David Ifeoluwa Adelani, and Jimmy Lin. 2023. Better Quality Pre-training Data and T5 Models for African Languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pr… find
  39. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. find our card
  40. Maja Pavlovic and Massimo Poesio. 2024. The effectiveness of LLMs as annotators: A comparative overview and empirical analysis of direct representation. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024, pages 100–110, Torino, Italia. ELRA and… find
  41. Qwen Team. 2024. Qwen2.5: A Party of Foundation Models. find
  42. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michał Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-text Transformer. Journal of Machine Learning Research, 21(140):1–67. find
  43. Surangika Ranathunga, Nisansa de Silva, Menan Velayuthan, Aloka Fernando, and Charitha Rathnayake. 2024. Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational L… find
  44. Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. 2021. ”Everyone Wants to do the Model Work, Not the Data Work”: Data Cascades in High-Stakes AI. Proceedings of CHI Conference on Human Factors in Computing Systems (CHI ’21), pages 1–15. find
  45. Marco Siino, Ilenia Tinnirello, and Marco La Cascia. 2024. Is Text Preprocessing Still Worth the Time? A Comparative Survey on the Influence of Popular Preprocessing Methods on Transformers and Traditional Classifiers. Information Systems, 121. find
  46. Lilli Smal, Andrea Lösch, Josef van Genabith, Maria Giagkou, Thierry Declerck, and Stephan Busemann. 2020. Language Data Sharing in European Public Services – Overcoming Obstacles and Creating Sustainable Data Sharing Infrastructures. In Proceedings of the Twelfth Language Resources and Evaluation C… find
  47. Kushal Tatariya, Artur Kulmizev, Wessel Poelman, Esther Ploeger, Marcel Bollmann, Johannes Bjerva, Jiaming Luo, Heather Lent, and Miryam de Lhoneux. 2025. How Good is Your Wikipedia? Auditing Data Quality for Low-resource and Multilingual NLP. find
  48. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2025. Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM 2 ), pages 404–430. find
  49. Khanh-Tung Tran, Barry O’Sullivan, and Hoang Nguyen. 2024a. Irish-based Large Language Model with Extreme Low-Resource Settings in Machine Translation. In Proceedings of the Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2024), pages 193–202, Bangkok, Tha… find
  50. Khanh-Tung Tran, Barry O’Sullivan, and Hoang D. Nguyen. 2024b. UCCIX: Irish-eXcellence Large Language Model. find
  51. Abigail Walsh, Órla Ní Loinsigh, Jane Adkins, Ornait O’Connell, Mark Andrade, Teresa Clifford, Federico Gaspari, Jane Dunne, and Brian Davis. 2025. eSTÓR: Curating Irish Datasets for Machine Translation. In Proceedings of Machine Translation Summit XX Volume 2, page 115–116, Geneva, Switzerland. Eur… find
  52. Language Resource References Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and Stephan Vogel. 2014. The AMARA Corpus: Building Parallel Language Resources for the Educational Domain. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1044–10… find our card
  53. Allen AI. 2023. Allen AI: Irish dataset. Accessed: 25 June 2025. find
  54. Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume… find our card
  55. Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, Jindřich Helcl, Erik Henriksson, Mateusz Klimaszewski, Ville Komulainen, Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Petter Mæhlum, Bhavitvya Mal… find our card
  56. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No Language Left Behind: Scaling Human-Centered Machine Translation. find
  57. Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema Ramírez-Sánchez, Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen, and Jörg Tiedemann. 2024. A New Massive Multilingual Dataset for High-Performance Language Technolo… find
  58. Ahmed El-Kishky, Adithya Renduchintala, James Cross, Francisco Guzmán, and Philipp Koehn. 2021. XLEnt: Mining a Large Cross-Lingual Entity Dataset with Lexical-Semantic-Phonetic Word Alignment. arXiv preprint arXiv:2104.08597. find
  59. Nathan Godey. 2023. OSCAR Small: Unshuffled and Deduplicated Irish Dataset. Accessed: 25 June 2025. find
  60. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry,… find our card
  61. Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923–929. find our card
  62. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages. arXiv preprint arXiv:2309.09400. find our card
  63. OSCAR Project. 2024. OSCAR: Open Source Project on Multilingual Resources for Machine Learning. Accessed: 25 June 2025. find
  64. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand Joulin. 2019. CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web. arXiv preprint arXiv:1911.04944. find our card
  65. Jörg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA). find
  66. Wikimedia Foundation. 2023. Wikipedia Language Edition: Irish (ga), Dump from 1 November 2023. Accessed: 25 June 2025. 11. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/walsh-2026-quality-appropriateness-large-text-datasets/