Elamur Research⌕
☰
home / papers / 2024 / UCCIX: Irish-eXcellence Large Language Model

UCCIX: Irish-eXcellence Large Language Model

Paper · 2024 · Khanh-Tung Tran, Barry O'Sullivan and Hoang D. Nguyen

Summary

Irish is a UNESCO "definitely endangered" language with only a few hundred million tokens of available text, far below what scaling laws suggest for training an LLM. The authors build UCCIX, an open-source Irish LLM, by continued pre-training of Llama 2-13B. They selected that base by its perplexity on Irish. They gathered and cleaned the Irish open-source data they could find, removed duplicates across sources with 5-gram matching, and added 10k Irish tokens to the tokenizer. Training first used English–Irish parallel sentences, then a much larger monolingual Irish corpus. An instruction-tuned variant, UCCIX-Instruct, followed. Because Irish lacks generative benchmarks, they also built IrishQA and an Irish version of MT-bench. UCCIX beats much larger models on Irish tasks, by up to 12 points on the Cloze Test. Its performance on English benchmarks drops, which the authors attribute to catastrophic forgetting. The work matters as a low-data recipe that could transfer to other endangered or indigenous languages, including other low-resource settings.

Key points

  • Pre-training corpus: about 3.1B characters before cleaning and 1.8B after. Sources are CulturaX-ga (65%), Glot500-ga (27.1%), the Irish side of ParaCrawl v7, Irish Wikipedia and Corpora Irish. ELRC parallel data, about 1% of the corpus size, is scheduled first.
  • Base model choice: Llama 2-13B had the lowest Irish perplexity (8.94), ahead of Mistral-7B (11.68) and BLOOM-7B (63.75). Vocabulary grew from 32k to 42k tokens.
  • Irish results: Cloze Test accuracy 0.75 vs 0.63 for Llama 2-70B; SIB-200 (Irish) 0.78 vs 0.71; IrishQA (ga) exact match 0.39 vs 0.31. Ablations show that both the parallel-data stage and the vocabulary expansion help.
  • Tokenizer efficiency: about 3.47 chars/token vs 2.26 for Llama 2, roughly 1.54× faster generation. Fertility fell from 1.95 to 1.57 tokens/word.
  • New resources: IrishQA has 60 context-grounded QA pairs in English and Irish, written by native speakers. MT-bench was machine-translated and then corrected by native speakers. On Irish MT-bench, UCCIX-Instruct scores 5.01, close to gpt-3.5-turbo's 5.47, while Llama 2-13B-chat scores 1.94.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University College CorkCork · Ireland6 works · 119 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Khanh-Tung Tran, Barry O'Sullivan and Hoang D. Nguyen. 2024. UCCIX: Irish-eXcellence Large Language Model. arXiv:2405.13010.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.13010 · arXiv 2405.13010

@misc{tran2024uccix,
  title = {{UCCIX: Irish-eXcellence Large Language Model}},
  author = {Khanh-Tung Tran, Barry O'Sullivan and Hoang D. Nguyen},
  year = {2024},
  doi = {10.48550/arxiv.2405.13010},
  eprint = {2405.13010},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.13010}
}

References

14 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cloud Translation — cloud.google.com. https://cloud.google.com/ translate?hl=en. [Accessed 09-05-2024]. ↗
  2. David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, and En-Shiun Lee, ‘SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects’, in Proceedings of the 18th Conference of the European Chapter of the…
  3. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck, ‘Llemma: An open language model for mathematics’, in The Twelfth International Conference on Learning Representations, (2024).
  4. Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume… — ParaCrawl: Web-Scale Acquisition of Parallel Corpora
  5. James Barry, Joachim Wagner, Lauren Cassidy, Alan Cowap, Teresa Lynn, Abigail Walsh, Mícheál J. Ó Meachair, and Jennifer Foster, ‘gaBERT — an Irish language model’, in Proceedings of the Thirteenth Language Resources and Evaluation Conference, eds., Nicoletta Calzolari, Frédéric Béchet, Philippe Bla… find
  6. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie, ‘A survey on evaluation of large language models’, ACM Trans. Intell. Syst. Technol., 15(3), (mar 2024).
  7. European Language Resource Coordination, AI for Multilingual Europe – Why Language Data Matters, ELRC Consortium, 2022. find
  8. Leo Gao et al. A framework for few-shot language model evaluation, 12 2023. find
  9. Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff, ‘Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages’, in Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), eds., Nicoletta Calzolari, Khalid Choukri, Th…
  10. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi… find
  11. Jordan Hoffmann et al., ‘An empirical analysis of compute-optimal large language model training’, in Advances in Neural Information Processing Systems, eds., Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, (2022). find
  12. Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze, ‘Glot500: Scaling multilingual corpora and language models to 500 languages’, in Proceedings of the 61st Annual M… find
  13. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut find
  14. Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed, ‘Mistral 7b’, CoRR, abs/2310.06825, (2023). Seamus Lankford, Haithem Afli, and Andy Way, ‘Machine translation in the covid domain: an English-Irish case study for LoResMT 2021’, in Proceedings of the 4th Workshop on Technologies for MT of… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tran-2024-uccix-irish-excellence-large-language/