Elamur Research⌕
☰
home / papers / 2024 / CUFE at NADI 2024 shared task: Fine-Tuning Llama-3 To Translate From Arabic Dialects To Modern Standard Arabic

CUFE at NADI 2024 shared task: Fine-Tuning Llama-3 To Translate From Arabic Dialects To Modern Standard Arabic

Paper · 2024 · Michael Ibrahim

Summary

This system paper from the NADI 2024 shared task (DA-MSA machine translation subtask) addresses translating sentences from four Arabic dialects (Egyptian, Emirati, Jordanian, Palestinian) into Modern Standard Arabic (MSA). Dialectal Arabic is low-resource, and general LLMs handle it poorly. The author adapts Llama-3 8B with parameter-efficient fine-tuning: the base weights are frozen and quantized to 4 bits, and only LoRA matrices are trained. Each training example is an instruction prompt that names the source dialect and casts the model as an expert Arabic proofreader. Two training sources are compared. One is the OSACT 2024 development set, which covers different dialect labels. The other is the small NADI 2024 development set, which matches the test dialects. Training on the 400 matched NADI sentences gives a test BLEU of 21.44. Training on the 1001-example OSACT set gives 16.09. The whole run took about an hour on a single Colab GPU. The work shows that cheap adapter tuning on a small, distribution-matched set can turn an open LLM into a workable dialect normalizer.

Key points

  • Base model and adapters: Llama-3 8B with LoRA (rank 64, scaling factor 16, dropout 0.05) applied to all linear projections (q/k/v/o, gate, up, down), with 4-bit weight quantization and fp16/fp32 mixed precision.
  • Training settings: Adam with learning rate 1e-4, batch size 1, evaluation every 50 steps, a single NVIDIA A100 40GB on Google Colab, roughly one hour including test generation.
  • OSACT 2024 data: 1001 pairs split across Egyptian, Maghrebi, Levantine, Gulf and Iraqi (about 200 each). NADI 2024 data: a dev set of 400 sentences (100 per dialect) and a test set of 2000 sentences (500 per dialect).
  • Results: OSACT-trained, the system scores 15.52 BLEU on NADI dev and 16.09 on NADI test. NADI-dev-trained, it scores 21.44 BLEU on NADI test.
  • OSACT has no Emirati, Jordanian or Palestinian labels, so at inference Emirati was mapped to Gulf, and Jordanian and Palestinian to Levantine.

Interesting ideas

  • I1. Putting the source variety's name into the instruction lets one model serve several dialects, and unseen fine-grained labels can fall back to the nearest coarser label seen in training.
  • I2. A small set drawn from the target distribution (400 sentences) beat a set more than twice as large whose labels did not match the test.
  • I3. The related work frames dialect-to-standard conversion as a preprocessing step, so that tools built for the standard variety can then be applied.
  • I4. 4-bit quantization combined with LoRA makes adapting an 8B model a one-GPU, one-hour job.
  • I5. Running the same pipeline on two different training sources and scoring both on one shared test gives a clean way to isolate the effect of the data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could take a genre/variety tag (web, spoken, legal), and treebanks with unseen genres could be mapped to the closest trained tag instead of being treated as unknown.
  • I2 (learn): when choosing silver or teacher data, measure how well each source matches the target domain, and test small in-domain sets against large mismatched ones before scaling up.
  • I3 (tokens): a normalizer for informal English (slang spellings, social-media forms) could run before the tagger, with an alignment kept back to the original tokens so every step stays explainable.
  • I4 (learn): a cheap tier of task-specific LLM teachers could be fine-tuned this way to produce candidate annotations, which the witness parsers would then vote on.
  • I5 (learn): evaluation reports could hold the model fixed and vary only the training data, which works as a control showing what the data alone contributes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cairo UniversityGiza · Egypt42 works · 1,350 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Michael Ibrahim. 2024. CUFE at NADI 2024 shared task: Fine-Tuning Llama-3 To Translate From Arabic Dialects To Modern Standard Arabic. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.87

@inproceedings{ibrahim2024cufe,
  title = {{CUFE at NADI 2024 shared task: Fine-Tuning Llama-3 To Translate From Arabic Dialects To Modern Standard Arabic}},
  author = {Michael Ibrahim},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.87},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.87}
}

References

19 references, 11 of them with a card of ours. ↗ leads to the work.

  1. Muhammad Abdul-Mageed, Amr Keleg, AbdelRahim Elmadany, Chiyu Zhang, Injy Hamed, Walid Magdy, Houda Bouamor, and Nizar Habash. 2024. NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024).
  2. Abidrabbo Alnassan. 2023. Automatic standardization of arabic dialects for machine translation. arXiv preprint arXiv:2301.03447. ↗
  3. Hanin Atwany, Nour Rabih, Ibrahim Mohammed, Abdul Waheed, and Bhiksha Raj. 2024. Osact 2024 task 2: Arabic dialect to msa translation. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation@ LREC-COLING 2024, pages 98–103.
  4. Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. 2023. Parameter-efficient fine-tuning design spaces. arXiv preprint arXiv:2301.01821. ↗
  5. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. ↗
  6. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210. ↗
  7. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR. ↗
  8. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. ↗
  9. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. ↗
  10. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. ↗
  11. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043. ↗
  12. Laith H Baniata, Seyoung Park, and Seong-Bae Park. 2018. A neural machine translation model for arabic dialects that utilizes multitask learning (mtl). Computational Intelligence & Neuroscience. ↗
  13. Séamus Lankford, Haithem Afli, and Andy Way. 2023. adaptmllm: Fine-tuning multilingual language models on low-resource languages with integrated llm playgrounds. Information, 14(12):638. ↗
  14. Rachel Bawden and François Yvon. 2023. Investigating the translation performance of a large multilingual language model: the case of bloom. arXiv preprint arXiv:2303.01911. ↗
  15. Atul Kr Ojha, Chao-Hong Liu, Katharina Kann, John Ortega, Sheetal Shatam, and Theodorus Fransen. 2021. Findings of the loresmt 2021 shared task on covid and sign language for low-resource languages. arXiv preprint arXiv:2108.06598. ↗
  16. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot Wael Salloum and Nizar Habash. 2013. Dialectal arabic to english machine translation: Pivoting through modern standard arabic. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 348–358.
  17. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  18. Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. 2024. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data, 18(6):1–32. ↗
  19. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Multilingual machine translation with large language models: Empirical results and analysis. arXiv preprint arXiv:2304.04675. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ibrahim-2024-cufe-nadi-2024-shared-task/