Elamur Research⌕
☰
home / papers / 2025 / Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks

Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks

Paper · 2025 · Maitha Alshehhi, Ahmed Sharshar and Mohsen Guizani

Summary

The paper asks how well current open LLMs handle languages with fewer resources, and what happens to that ability when the models are compressed. Six decoder-only model families were tested: BLOOMZ, AceGPT, Jais, LLaMA-2, XGLM and AraGPT2. Where available, both a small and a large size of each family were used. The test sets were ArabicMMLU, EnglishMMLU and the Indic benchmark Kannada-ARC-C-2.5K. Each model was also run in compressed variants: 4-bit and 8-bit quantization, and pruning at 20%, 40% and 80%. Accuracy was reported per subject category. Broad multilingual models generally beat language-specific ones, which points to useful transfer across languages; AraGPT2 in particular trails. Quantization usually keeps accuracy close to the full model. Heavy pruning, by contrast, badly damages results, especially for the larger models. The work matters because real deployments in underserved languages often need small, cheap models. It shows which compression settings are safe to use and which ones quietly break the model.

Key points

  • The models and sizes tested were:
    • BLOOMZ 560M/7B
    • AceGPT 7B/13B
    • Jais 13B
    • LLaMA-2 7B/13B
    • XGLM 1.7B/7.5B
    • AraGPT2 135M/1.46B
  • Each model was compressed into a fixed grid of variants: 4Q, 8Q, 20P, 40P and 80P. Jais appears only in pruned variants in the tables.
  • On EnglishMMLU, BLOOMZ-7.1B reaches about 51 in Social Sciences, but its 80%-pruned variant falls to about 23. AceGPT-13B scores about 48 in Humanities, and its 80% pruning collapses to under 9.
  • On ArabicMMLU, the best scores sit in the low-to-mid 40s. Examples are BLOOMZ-7.1B on Language (about 44.5) and Jais-13B (about 43.7, rising slightly at 40% pruning). AraGPT2-1.47B drops to single digits on several EnglishMMLU categories.
  • Quantization is not always harmless. AceGPT-7B loses roughly ten points in English STEM under 4-bit quantization.

Interesting ideas

  • I1. Applying the same ladder of compression levels to every model turns compression into a sweep with a visible breaking point, rather than a single yes/no test.
  • I2. Comparing the smallest and largest member of each model family helps separate the effect of scale from the effect of architecture and training recipe.
  • I3. Breaking accuracy down by subject category shows that compression and language effects are uneven, and an average would hide this. Pruning, for example, hurts Humanities much more for some models.
  • I4. Many heavily compressed variants score around the level of random guessing on four-choice questions. Very low numbers are a sign of a broken model, not of a weak one.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): run gparse through the same quantization and pruning ladder and track LAS at each step. This finds the cheapest setting before accuracy drops off sharply.
  • I2 (learn): when comparing parser or tagger variants, train each in two sizes, so a gain from the method is not confused with a gain from capacity.
  • I3 (answer): report school-course test results per subject and per task type. A regression in one area then cannot hide behind a stable overall score.
  • I4 (learn): add a random-answer baseline as a negative control to every multiple-choice evaluation. Any system that does not clearly beat this baseline is then flagged automatically.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Maitha Alshehhi, Ahmed Sharshar and Mohsen Guizani. 2025. Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks. arXiv:2507.19699.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2507.19699 · arXiv 2507.19699

@misc{alshehhi2025inclusive,
  title = {{Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks}},
  author = {Maitha Alshehhi, Ahmed Sharshar and Mohsen Guizani},
  year = {2025},
  doi = {10.48550/arxiv.2507.19699},
  eprint = {2507.19699},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.19699}
}

References

3 references, 3 of them with a card of ours. ↗ leads to the work.

  1. 1. Almazrouei, E., Cojocaru, R., Baldo, M., Malartic, Q., Alobeidli, H., Mazzotta, D., Penedo, G., Campesan, G., Farooq, M., Alhammadi, M., et al.: Alghafa evaluation benchmark for arabic language models. In: Proceedings of ArabicNLP 2023. pp. 244–275 (2023) 2. Antoun, W., Baly, F., Hajj, H.: Aragpt2: Pre-trained transformer for arabic language generation. arXiv preprint arXiv:2012.15520 (2020) 3. B, S., P, P.R., B, S.M., S, K.: The evolution of large language model: Models, applications and challenges pp. 1–8 (2024). https://doi.org/10.1109/ICCTAC61556.2024.10581180 4. Bari, M.S., Alnumay, Y., Alzahrani, N.A., Alotaibi, N.M., Alyahya, H.A., Al-Rashed, S., Mirza, F.A., Alsubaie, S.Z., Alahmed, H.A., Alabduljabbar, G., et al.: Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390 (2024) 5. Boughorbel, S., Hawasly, M.: Analyzing multilingual competency of llms in multi-turn instruction following: A case study of arabic. arXiv preprint arXiv:2310.14819 (2023) 6. Cheung, M.: A reality check of the benefits of llm in business. arXiv preprint arXiv:2406.10249 (2024) 7. Chu, Z., Wang, S., Xie, J., Zhu, T., Yan, Y., Ye, J., Zhong, A., Hu, X., Liang, J., Yu, P.S., et al.: Llm agents for education: Advances and applications. arXiv preprint arXiv:2503.11733 (2025) 8. Dey, K., Tarannum, P., Hasan, M.A., Razzak, I., Naseem, U.: Better to ask in english: Evaluation of large language models on english, low-resource and cross-lingual settings. arXiv preprint arXiv:2410.13153 (2024) 9. Hagos, D.H., Battle, R., Rawat, D.B.: Recent advances in generative ai and large language models: Current status, challenges, and perspectives. IEEE Transactions on Artificial Intelligence 5(12), 5873–5893 (2024). https://doi.org/10.1109/TAI.2024.3444742 10. Hasan, M.A., Tarannum, P., Dey, K., Razzak, I., Naseem, U.: Do large language models speak all languages equally? a comparative study in low-resource settings. arXiv preprint arXiv:2408.02237 (2024) 11. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., Steinhardt, J.: Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020) 12. Hijazi, F., AlHarbi, S., AlHussein, A., Shairah, H.A., AlZahrani, R., AlShamlan, H., Knio, O., Turkiyyah, G.: Arablegaleval: A multitask benchmark for assessing arabic legal knowledge in large language models. arXiv preprint arXiv:2408.07983 (2024) 13. Huang, H., Yu, F., Zhu, J., Sun, X., Cheng, H., Song, D., Chen, Z., Alharthi, A., An, B., He, J., et al.: Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053 (2023) 14. Indic-Benchmark: Indic-benchmark/kannada-arc-c-2.5k. https://huggingface.co/datasets/Indic-Benchmark/kannada-arc-c-2.5k (2023) 15. Koto, F., Li, H., Shatnawi, S., Doughman, J., Sadallah, A.B., Alraeesi, A., Almubarak, K., Alyafeai, Z., Sengupta, N., Shehata, S., et al.: Arabicmmlu: Assessing massive multitask language understanding in arabic. arXiv preprint arXiv:2402.12840 (2024) ↗
  2. Towards Inclusive NLP 16. Liang, M., Arun, A., Wu, Z., Munoz, C., Lutch, J., Kazim, E., Koshiyama, A., Treleaven, P.: Thames: An end-to-end tool for hallucination mitigation and evaluation in large language models. arXiv preprint arXiv:2409.11353 (2024) 17. Lin, X.V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., et al.: Few-shot learning with multilingual language models. arXiv preprint arXiv:2112.10668 (2021) 18. Nafea, Y., Shehata, S., Talat, Z., Aboeitta, A., Sharshar, A., Nakov, P.: Araoffence: Detecting offensive speech across dialects in arabic media. In: Proceedings of Interspeech 2024. pp. 4303–4307 (2024). https://doi.org/10.21437/Interspeech.2024-2077 19. Narayanan, V., KP, P.R., Nouphal, S.: Suvach–generated hindi qa benchmark. arXiv preprint arXiv:2404.19254 (2024) 20. Nazi, Z.A., Peng, W.: Large language models in healthcare and medical domain: A review. In: Informatics. vol. 11, p. 57. ↗
  3. MDPI (2024) 21. Plaza, I., Melero, N., del Pozo, C., Conde, J., Reviriego, P., Mayor-Rocher, M., Grandury, M.: Spanish and llm benchmarks: is mmlu lost in translation? arXiv preprint arXiv:2406.17789 (2024) 22. Refai, D., Abu-Soud, S., Abdel-Rahman, M.J.: Data augmentation using transformers and similarity measures for improving arabic text classification. IEEE Access 11, 132516–132531 (2023) 23. Sengupta, N., Sahu, S.K., Jia, B., Katipomu, S., Li, H., Koto, F., Marshall, W., Gosal, G., Liu, C., Chen, Z., et al.: Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149 (2023) 24. Sibaee, S.T., Alharbi, A.I., Ahmed, S., Nacar, O., Ghouti, L., Koubaa, A.: Asos at arabic llms hallucinations 2024: Can llms detect their hallucinations. In: Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation@ LREC-COLING 2024. pp. 130–134 (2024) 25. Singh, H., Gupta, N., Bharadwaj, S., Tewari, D., Talukdar, P.: Indicgenbench: a multilingual benchmark to evaluate generation capabilities of llms on indic languages. arXiv preprint arXiv:2404.16816 (2024) 26. Taghanaki, S.A., Khani, A., Khasahmadi, A.: Mmlu-pro+: Evaluating higher-order reasoning and shortcut learning in llms. arXiv preprint arXiv:2409.02257 (2024) 27. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023) 28. Wang, Z., Wohlwend, J., Lei, T.: Structured pruning of large language models. arXiv preprint arXiv:1910.04732 (2019) 29. Workshop, B., Scao, T.L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A.S., Yvon, F., et al.: Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 (2022) 30. Yao, Z., Li, C., Wu, X., Youn, S., He, Y.: A comprehensive study on post-training quantization for large language models. arXiv preprint arXiv:2303.08302 (2023) — ASOS at Arabic LLMs Hallucinations 2024: Can LLMs detect their Hallucinations :) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alshehhi-2025-towards-inclusive-nlp-assessing-compressed/