Elamur Research⌕
☰
home / papers / 2026 / Multi-Task Instruction Tuning via Data Scheduling for Low-Resource Arabic SpeechLLMs

Multi-Task Instruction Tuning via Data Scheduling for Low-Resource Arabic SpeechLLMs

Summary

The paper asks how the ordering and batching of training data affect multi-task instruction tuning of an audio LLM for Arabic–English speech. In this setting, very large ASR corpora tend to overwhelm small paralinguistic datasets and rare labels. The authors adapt Qwen2.5-Omni-7B as their main model and Gemma-4-E4B-it as a contrastive check, both with LoRA. The tasks cover ASR, summarization of both text and speech, determining which dialect is spoken (DID), and classifying the emotion conveyed in speech (SER).

At an equal step budget they compare four schedules:

  • uniform mixing (UM),
  • a task-progressive curriculum (TPC),
  • aligner-based diverse sampling (ADS), which builds batches from clusters of aligner embeddings,
  • a two-stage TPC→ADS.

They also release AraMega-SSum, a new semi-synthetic resource for Arabic speech summarization.

TPC is strongest on generative tasks. ADS used alone helps paralinguistic tasks but makes generation less stable. TPC→ADS gives the best DID and SER and beats Gemini-2.5-Pro on both. The work shows that careful data scheduling can partly offset small model size in dialect-rich, low-resource conditions.

Key points

  • AraMega-SSum: 49,995 training clips (222 h) and 4,000 test clips (16.98 h).
    • Built by translating MegaSSUM summaries into MSA and synthesizing speech conditioned on 47 speakers from 10 Arab countries.
    • Part of it was checked by humans.
  • Two-phase training:
    • Phase 1 tunes the audio encoder and aligner on bilingual ASR, with LoRA on the LLM.
    • Phase 2 freezes both and updates only LoRA under each schedule.
  • TPC order: ASR first, then DID/SER, then TSUM/SSUM, keeping a share of earlier-stage data at each step.
  • ADS construction:
    • Max-pools aligner states and runs K-Means with K=500 on a 3% subset.
    • Fills batches by task proportions, with upsampling of minority labels.
    • Walks round-robin across clusters.
  • Results:
    • TPC→ADS reaches DID 88.8 vs 63.4 and SER 75.94 vs 59.3 against Gemini-2.5-Pro.
    • On Qwen, its SER is +16.6 over TPC.
    • For Gemma, plain UM gave the best ASR, likely because of Gemma's prior Arabic exposure.

Interesting ideas

  • I1. Discretize an intermediate representation (here the aligner output) with K-Means, then sample round-robin over clusters inside each task-label cell, so every update sees varied conditions instead of whatever dominates the corpus. — For Mova (learn): Cluster silver-data sentences by gparse encoder vectors and draw training batches across clusters, so rare constructions are not drowned out by frequent newswire-style sentences.
  • I2. Order training from low-level to high-level objectives, keeping a slice of earlier data as replay, and only then switch to diversity-oriented sampling. Stabilizing shared representations first seems to be what lets the diversity phase pay off. — For Mova (tree): Train gparse first on tagging/morphology signals, then on arcs and labels with replay, and add the diversity-balanced phase last.
  • I3. Compare training schedules at an identical total step budget, so any gap can be credited to sampling and not to extra compute. — For Mova (learn): Run every curriculum or silver-data mixing experiment at fixed steps, alongside negative controls, so claimed gains are not budget artifacts.
  • I4. Rerun the study on a second architecture with different priors as a sensitivity check. Here the curriculum effect on ASR vanished for the model that already knew Arabic. — For Mova (learn): Test each scheduling finding on both the perceptron parser and gparse before adopting it, since a stronger prior may make the trick unnecessary.
  • I5. Cast classification as emitting a canonical label string, so discriminative and generative tasks share one loss and one interface. — For Mova (frames): When LLM teacher tiers label speech acts or irony, require a fixed canonical label vocabulary so outputs can be checked mechanically and merged with other witnesses.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2601.12494 · arXiv 2601.12494

@misc{bhatti2026multi,
  title = {{Multi-Task Instruction Tuning via Data Scheduling for Low-Resource Arabic SpeechLLMs}},
  author = {Bhatti, Hunzalah Hassan and Alam, Firoj and Chowdhury, Shammur Absar},
  year = {2026},
  doi = {10.48550/arxiv.2601.12494},
  eprint = {2601.12494},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2601.12494}
}

References

44 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammad Al-Fetyani, Muhammad Al-Barham, Gheith Abandah, Adham Alsharkawi, and Maha Dawas. 2023. MASC: Massive arabic speech corpus. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 1006--1013. Firoj Alam, Ali Ezzat Shahroor, Md. find
  2. Arid Hasan, Zien Sheikh Ali, Hunzalah Hassan Bhatti, Mohamed Bayan Kmainasi, Shammur Absar Chowdhury, Basel Mousi, Fahim Dalvi, Nadir Durrani, and Natasa Milic-Frayling. 2025. OA-SIS: A multilingual and multimodal framework for culturally grounded spoken visual qa. arXiv preprint arXiv:2510.06371. ↗ our card
  3. Sadeen Alharbi, Areeb Alowisheq, Zoltán Tüske, Kareem Darwish, Abdullah Alrajeh, Abdulmajeed Alrowithi, Aljawharah Bin Tamran, Asma Ibrahim, Raghad Aloraini, Raneem Alnajim, Ranya Alkahtani, Renad Almuasaad, Sara Al-rasheed, Shaykhah Alsubaie, and Yaser Alonaizan. 2024. SADA: Saudi audio dataset for arabic. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10286--10290. find
  4. Ahmed Ali, Peter Bell, James Glass, Yacine Messaoui, Hamdy Mubarak, Steve Renals, and Yifan Zhang. 2016. The MGB-2 challenge: Arabic multi-dialect broadcast media recognition. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 279--284. find our card
  5. Ahmed Ali, Suwon Shon, Younes Samih, Hamdy Mubarak, Ahmed Abdelali, James Glass, Steve Renals, and Khalid Choukri. 2019. The MGB challenge: Recognition and dialect identification of dialectal arabic speech. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1026--1033. IEEE. find
  6. Ahmed Ali, Stephan Vogel, and Steve Renals. 2017. Speech recognition challenge in the wild: Arabic mgb-3. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 316--322. find our card
  7. Zien Sheikh Ali, Hunzalah Hassan Bhatti, Rabindra Nath Nandi, Shammur Absar Chowdhury, and Firoj Alam. 2026. MENASpeech-Bank: A reference voice bank with persona-conditioned multi-turn conversations for audiollms. arXiv preprint arXiv:2602.07036. ↗
  8. Sara Althubaiti, Vasista Sai Lodagala, Tjad Clark, Yousseif Ahmed Elshahawy, Daniel Izham, Abdullah Alrajeh, Aljawahrah Bin Tamran, and Ahmed Ali. 2025. Octopus: Towards building the Arabic speech LLM suite. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 425--435, Suzhou, China. Association for Computational Linguistics. find
  9. Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. 2022. SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5723--5738, Dublin, Ireland. Association for Computational Linguistics. find
  10. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common Voice: A massively-multilingual speech corpus. In Proceedings of the twelfth language resources and evaluation conference, pages 4218--4222. find our card
  11. Mourad Belhadj, Ilham Bendellali, and Elalia Lakhdari. 2022. KEDAS: A validated arabic speech emotion dataset. In 2022 International Symposium on iNnovative Informatics of Biskra (ISNIB), pages 1--6. find
  12. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning (ICML), pages 41--48. ACM. find our card
  13. Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, and Julian Weber. 2024. XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model. In Interspeech 2024, pages 4978--4982. find
  14. Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Zhao You, and Zhiyong Yan. 2021. GigaSpeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. In Interspeech 2021. ISCA. find
  15. Zeren Chen, Huayang Liu, Zhenfei Yin, Si Liu, Lyu Sheng, Wanli Ouyang, Jing Shao, and 1 others. 2024. Octavius: Mitigating task interference in mllms via lora-moe. In International Conference on Learning Representations, volume 2024, pages 24652--24673. find
  16. Shammur A. Chowdhury, Younes Samih, Mohamed Eldesouki, and Ahmed Ali. 2020. Effects of Dialectal Code-Switching on Speech Modules: A Study Using Egyptian Arabic Broadcast Speech. In Interspeech 2020, pages 2382--2386. find
  17. Shammur Absar Chowdhury, Amir Hussein, Ahmed Abdelali, and Ahmed Ali. 2021. Towards One Model to Rule All: Multilingual Strategy for Dialectal Code-Switching Arabic ASR. In Proc. of INTERSPEECH. find our card
  18. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-Audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919. Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, and 1 others. 2025. CosyVoice 3: Towards in-the-wild speech generation via scaling-up and post-training. arXiv preprint arXiv:2505.17589. ↗ our card
  19. Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29--48. find
  20. François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Estève. 2018. TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation, page 198208. Springer International Publishing. find
  21. Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiang-Yang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, and Sheng Zhao. 2024. Naturalspeech 3: zero-shot speech synthesis with factorized codec and diffusion models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. find
  22. Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022. AraBART: a pretrained Arabic sequence-to-sequence model for abstractive summarization. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 31--42, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  23. Samira Klaylat, Ziad Osman, Rached Zantout, and Lama Hamandi. 2018. Arabic natural audio dataset. find
  24. Tom Ko, Vijayaditya Peddinti, Michael Seltzer, and Sanjeev Khudanpur. 2017. A study on data augmentation of reverberant speech for robust speech recognition. pages 5220--5224. find
  25. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics. find
  26. Kohei Matsuura, Takanori Ashihara, Takafumi Moriya, Masato Mimura, Takatomo Kano, Atsunori Ogawa, and Marc Delcroix. 2024. Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillation. In Interspeech 2024, pages 1945--1949. find
  27. Ali Hamid Meftah, Yousef Ajami Alotaibi, and Sid-Ahmed Selouani. 2018. Evaluation of an arabic speech corpus of emotions: A perceptual and statistical analysis. IEEE Access, 6:72845--72861. find
  28. Ali Hamid Meftah, Mustafa A. Qamhan, Yasser Seddiq, Yousef A. Alotaibi, and Sid Ahmed Selouani. 2021. King saud university emotions corpus: Construction, analysis, evaluation, and comparison. IEEE Access, 9:54201--54219. find
  29. Hamdy Mubarak, Amir Hussein, Shammur Absar Chowdhury, and Ahmed Ali. 2021. QASR: QCRI aljazeera speech resource a large scale annotated Arabic speech corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2274--2285, Online. Association for Computational Linguistics. find
  30. Patrick K. ONeill, Vitaly Lavrukhin, Somshubra Majumdar, Vahid Noroozi, Yuekai Zhang, Oleksii Kuchaiev, Jagadeesh Balam, Yuliya Dovzhenko, Keenan Freyberg, Michael D. Shulman, Boris Ginsburg, Shinji Watanabe, and Georg Kucsko. 2021. SPGISpeech: 5,000 Hours of Transcribed Financial Audio for Fully Formatted End-to-End Speech Recognition. In Interspeech 2021, pages 1434--1438. find
  31. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206--5210. find
  32. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492--28518. PMLR. find our card
  33. Andrew Rouditchenko, Saurabhchand Bhati, Edson Araujo, Samuel Thomas, Hilde Kuehne, Rogerio Feris, and James Glass. 2025. Omni-R1: Do you really need audio to fine-tune your audio llm? arXiv preprint arXiv:2505.09439. ↗
  34. Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Padfield, James Qin, Danny Rozenberg, Tara Sainath, Johan Schalkwyk, Matt Sharifi, Michelle Tadmor Ramanovich, Marco Tagliasacchi, and 11 others. 2023. AudioPaLM: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925. ↗ our card
  35. Sarah Safwat, Mohammed Salem, and Nada Sharaf. 2023. Egyptian arabic emotional dataset (EAED). find
  36. Xiaojun Shan, Qi Cao, Xing Han, Haofei Yu, and Paul Pu Liang. 2025. MINT: Multimodal instruction tuning with multimodal interaction grouping. arXiv preprint arXiv:2506.02308. ↗
  37. Suwon Shon, Ahmed Ali, Younes Samih, Hamdy Mubarak, and James Glass. 2020. Adi17: A fine-grained arabic dialect identification dataset. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8244--8248. find our card
  38. David Snyder, Guoguo Chen, and Daniel Povey. 2015. MUSAN: A Multipurpose Corpus for Music and Noise. In arXiv preprint arXiv:1510.08484. ↗
  39. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025a. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215. ↗ our card
  40. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025b. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Figure 3: Emotion label distribution across training datasets. 2023a. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15757--15773, Singapore. Association for Computational Linguistics. ↗ our card
  41. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  42. Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, and 1 others. 2023b. Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037. ↗
  43. Guanlong Zhao, Sinem Sonsaat, Alif Silpachai, Ivana Lucic, Evgeny Chukharev-Hudilainen, John Levis, and Ricardo Gutierrez-Osuna. 2018. L2-ARCTIC: A Non-native English Speech Corpus. In Interspeech 2018, pages 2783--2787. find
  44. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging LLM-as-ajudge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36:46595--46623. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bhatti-2026-multi-task-instruction-tuning-data/