Elamur Research⌕
☰
home / papers / 2025 / Unicorn at NADI 2025 Subtask 3: GEMM3N-DR: Audio-Text Diacritic Restoration via Fine-tuning Multimodal Arabic LLM

Unicorn at NADI 2025 Subtask 3: GEMM3N-DR: Audio-Text Diacritic Restoration via Fine-tuning Multimodal Arabic LLM

Paper · 2025 · Lotfy Elrefai, Mohamed

Summary

The paper describes GEMM3N-DR, a second-place entry in NADI 2025 Subtask 3. The task is to restore full Arabic diacritics to spoken transcripts covering dialectal, MSA, Classical and code-switched speech, given both the undiacritized text and the matching audio. The author worked in the closed track, which allowed only the official training data. After filtering, that data was split into a gold-diacritized portion and an undiacritized "augment" portion. Unlabeled transcripts received pseudo-diacritics from the CATT model, and the audio was varied with nlpaug transformations. The instruction-tuned multimodal Gemma 3N was adapted with high-rank LoRA on attention, feed-forward and audio-specific projections. At inference it was prompted with Arabic instructions and seven examples. The final system reached 64% WER and 15% CER on the hidden test set. One notable finding is that the untuned model did worse when audio was added than with text alone, so the gains came from fine-tuning and prompting. The work shows that an end-to-end multimodal LLM can replace cascaded ASR-plus-text pipelines for this task.

Key points

  • Base model: gemma-3n-E4B-it with LoRA (r=128, α=16, dropout 0), trained via SFTTrainer. The best checkpoint was at step 16,500, and decoding used near-zero temperature (0.001) with up to 256 new tokens.
  • Data after dropping utterances under three words and stripping punctuation:
    • 51,517 train utterances (~88.9 h)
    • 6,087 augment utterances (~14.1 h)
    • 1,580 dev and 365 test utterances
  • Zero-shot test results were 71% WER / 23% CER with text only and 84% WER / 34% CER with text plus audio.
  • Fine-tuning results on test:
    • LoRA rank 8 for 5,000 steps: 82% WER.
    • Rank 128 plus 7-shot prompting: 64% WER / 15% CER.
  • On dev, 7 shots beat 3 shots (69.05 vs 73.21 WER; 20.84 vs 23.22 CER). The main errors were hallucinated or truncated outputs and unstable diacritics on rare patterns.

Interesting ideas

  • I1. Before trusting an extra input channel, compare it against a run without that channel; here the audio turned out to hurt the untuned model.
  • I2. A specialist model (CATT) labeled the unannotated part of the corpus, turning raw transcripts into extra supervised training data.
  • I3. Words not in the target script are detected and copied back unchanged at their original positions, which protects code-switched content from being mangled by the model.
  • I4. Few-shot examples were deliberately drawn from several varieties (MSA, Egyptian, Moroccan), and this diversity is credited with reducing inconsistent outputs.
  • I5. The error analysis separates hallucinated or incomplete outputs from wrong-but-complete ones, which points to checking that the output still matches the input skeleton.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Every new witness signal, such as an extra parser or a teacher tier, should be evaluated against a control run that omits it, so its contribution is proven rather than assumed.
  • I2 (learn): Silver trees or lemmas from a specialist component could fill gaps in sparse genres, with the source of each label recorded so it stays explainable.
  • I3 (tokens): The tagger and lemmatizer could mark foreign or out-of-alphabet spans as pass-through tokens, keeping alignment intact instead of forcing English morphology onto them.
  • I4 (learn): Prompts for LLM teacher tiers could include examples spanning genres such as fables, school tests and news, so silver annotations stay consistent across text types.
  • I5 (back): Generated or compressed text could be checked to confirm it covers every node of the meaning graph, flagging omissions and additions as distinct failure types.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ain Shams UniversityCairo · Egypt13 works · 109 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSemitic languages · 3,066 worksVision & language · 811 worksSpeech & spoken language · 519 works

Cite

Cite as Lotfy Elrefai, Mohamed. 2025. Unicorn at NADI 2025 Subtask 3: GEMM3N-DR: Audio-Text Diacritic Restoration via Fine-tuning Multimodal Arabic LLM. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.106

@inproceedings{mohamed2025unicorn,
  title = {{Unicorn at NADI 2025 Subtask 3: GEMM3N-DR: Audio-Text Diacritic Restoration via Fine-tuning Multimodal Arabic LLM}},
  author = {Lotfy Elrefai, Mohamed},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.106},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.106}
}

References

18 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Amine Ben Abdallah, Ata Kabboudi, Amir Kanoun, and Salah Zaiem. 2023. Leveraging data collection and unsupervised learning for code-switched tunisian arabic automatic speech recognition. Preprint, arXiv:2309.11327. ↗
  2. Full repository URL: https://github.com/ MohamedElrefai/GEMM3N-DR-NADI-2025-Subtask-3 ↗
  3. Maryam Khalifa Al Ali and Hanan Aldarmaki. 2024. Mixat: A data set of bilingual emirati-English speech. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024, pages 222–226, Torino, Italia. ELRA and ICCL. ↗
  4. Abdulmohsen Al-Thubaity, Atheer Alkhalifa, Abdulrahman Almuhareb, and Waleed Alsanie. 2020. Arabic diacritization using bidirectional long short-term memory neural networks with conditional random fields. IEEE Access, 8:154984–154996. ↗
  5. Faris Alasmary, Orjuwan Zaafarani, and Ahmad Ghannam. 2024. Catt: Character-based arabic tashkeel transformer. arXiv preprint arXiv:2407.03236. ↗
  6. Zeina Aldallal, Sara Chrouf, Khalil Hennara, Mohamed Motaism Hamed, Muhammad Hreden, and Safwan AlModhayan. 2025. Sadeed: Advancing arabic diacritization through small language model. arXiv preprint arXiv:2504.21635. ↗
  7. Hanan Aldarmaki and Ahmad Ghannam. 2023. Diacritic recognition performance in arabic asr. arXiv preprint arXiv:2302.14022. ↗
  8. Khalid Almeman, Mark Lee, and Ali Abdulrahman Almiman. 2013. Multi dialect arabic speech parallel corpora. In 2013 1st International Conference on Communications, Signal Processing, and their Applications (ICCSPA), pages 1–6. find
  9. Sara Shatnawi, Sawsan Alqahtani, and Hanan Aldarmaki. 2023. Automatic restoration of diacritics for speech data sets. arXiv preprint arXiv:2311.10771. ↗
  10. Bashar Talafha, Hawau Olamide Toyin, Peter Sullivan, AbdelRahim Elmadany, Abdurrahman Juma, Amirbek Djanibekov, Chiyu Zhang, Hamad Alshehhi, Hanan Aldarmaki, Mustafa Jarar, Nizar Habash, and Muhammad Abdul-Mageed. 2025. Nadi 2025: The first multidialectal arabic speech processing shared task. In The Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou. Association for Computational Linguistics. ↗
  11. Hawau Toyin, Rufael Marew, Humaid Alblooshi, Samar M. Magdy, and Hanan Aldarmaki. 2025. Ar-Voice: A Multi-Speaker Dataset for Arabic Speech Synthesis. In Interspeech 2025, pages 4808–4812. find
  12. Dimitra Vergyri and Katrin Kirchhoff. 2004. Automatic diacritization of arabic for acoustic modeling in speech recognition. ↗
  13. Injy Hamed, Ngoc Thang Vu, and Slim Abdennadher. 2020. ArzEn: A speech corpus for code-switched Egyptian Arabic-English. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4237–4246, Marseille, France. European Language Resources Association. find
  14. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  15. Yassine El Kheir, Hamdy Mubarak, Ahmed Ali, and Shammur Absar Chowdhury. 2024. Beyond orthography: Automatic recovery of short vowels and dialectal sounds in arabic. arXiv preprint arXiv:2408.02430. ↗
  16. Ajinkya Kulkarni, Atharva Kulkarni, Sara Abedal-mon’em Mohammad Shatnawi, and Hanan Aldarmaki. 2023. Clartts: An open-source classical arabic text-to-speech corpus. In 2023 INTERSPEECH, pages 5511–5515. *Figure 1: End-to-end pipeline for the diacritization system.* 1.2
  17. Ossama Obeid, Go Inoue, and Nizar Habash. 2022. Camelira: An arabic multi-dialect morphological disambiguator. arXiv preprint arXiv:2211.16807. ↗
  18. Arfath Pasha, Mohamed Al-Badrashiny, Mona T Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In Lrec, volume 14, pages 1094–1101.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elrefai-2025-unicorn-nadi-2025-subtask-3/