Elamur Research⌕
☰
home / papers / 2025 / ArabicWeb-Edu: Educational Quality Data for Arabic LLM Training

ArabicWeb-Edu: Educational Quality Data for Arabic LLM Training

Paper · 2025 · Majd Hawasly, Tasnim Mohiuddin, Hamdy Mubarak et al.

Summary

Arabic is heavily underrepresented in LLM training data, and the Arabic text that exists on the web is often noisy or repetitive. The authors bring the FineWeb-Edu recipe to Arabic. They drew 1 million Arabic pages from Common Crawl and had Qwen-2.5-72B-Instruct rate each page from 0 to 5 for educational value, using a zero-shot prompt adapted from FineWeb-Edu. These ratings were then distilled into a small classifier: a classification head trained on a frozen mGTE encoder. The classifier scores a large Arabic web corpus, and pages rated 0 or 1 are removed. The result is ArabicWeb-Edu.

To test the corpus, they trained two 7B bilingual Arabic-English models from scratch for about 800B tokens. The two runs differed only in whether the Arabic web portion was filtered. The filtered model scored better on most Modern Standard Arabic benchmarks, but it was mixed or slightly worse on dialectal tests. The study shows that educational filtering works for a low-resource language. It also shows that this filtering should be one part of a broader data mix rather than the whole strategy.

Key points

  • Seed labels: The 1M-document seed set has a skewed label distribution. About half the pages are class 2, a quarter class 1 and a fifth class 3, while class 5 is almost absent at 0.02%. The seed set is published on Hugging Face (sboughorbel/arabic-web-edu-seed).
  • Classifier: It uses mGTE (305M parameters, 8k-token context) with the encoder and embeddings frozen, so only the head is trained.
    • Training used 900k documents for 20 epochs at a learning rate of 3e-4.
    • On a 100k test split it reaches accuracy 0.63, weighted F1 0.61 and macro F1 0.38. Its F1 on class 5 is zero.
  • Threshold: The filter keeps classes 2–5. This is looser than FineWeb-Edu, which keeps only pages scored 3 or higher, and reflects how little top-quality Arabic content there is.
  • Ablation setup: Both models use the OLMo-7B architecture and an Arabic/English/code mix taken from Fanar. The baseline checkpoint at 826B tokens is compared with the Edu checkpoint at 841B tokens.
  • Benchmark results:
    • MSA gains: OALL-v2 rose from 31.50 to 34.69 (+10.13%) and ACVA from 49.10 to 55.28 (+12.59%).
    • Dialect results were mixed. ArabicMMLU/Lev improved by 22.79%, but PIQA/Lev fell by 3.65% and Belebele/Ar by 1.45%.

Interesting ideas

  • I1. An expensive LLM grades a sample once, and those grades train a lightweight head on a frozen encoder, which can then grade millions of items cheaply.
  • I2. The cut-off is chosen from the observed score distribution: when top-grade data is scarce, they drop only the worst classes rather than keeping only the best.
  • I3. The ablation changes exactly one ingredient of an otherwise identical training mix, then reports results on two separate slices (MSA and dialect). This exposes a cost that an average over all benchmarks would have hidden.
  • I4. A quality filter tuned toward formal, educational text quietly removes informal and dialectal material, so it is worth auditing what a filter discards, not only what it keeps.
  • I5. Reporting per-class precision and recall revealed that the rarest class collapses to zero F1, a failure that headline accuracy concealed.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): An LLM teacher tier could rate a sample of silver sentences for parse usefulness. A small head on gparse encodings could then extend those ratings to the full silver pool.
  • I2 (learn): Silver-data agreement thresholds could be set per source from each source's score histogram, rather than one fixed bar that leaves rare genres nearly empty.
  • I3 (learn): When changing one input, such as a treebank or a set of repair rules, evaluation should keep a separate "other register" slice as a control so regressions on it stay visible.
  • I4 (tree): Before filtering sentences out of treebank or silver pools, check whether conversational or nonstandard English is being lost, since that would weaken the parser on dialogue in fables and tales.
  • I5 (answer): Results on school tests and word problems should be broken down by question type, so that rare types the readers never get right show up explicitly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 works

Cite

Cite as Majd Hawasly, Tasnim Mohiuddin, Hamdy Mubarak and Sabri Boughorbel. 2025. ArabicWeb-Edu: Educational Quality Data for Arabic LLM Training. Proceedings of The Third Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-main.36

@inproceedings{hawasly2025arabicweb,
  title = {{ArabicWeb-Edu: Educational Quality Data for Arabic LLM Training}},
  author = {Majd Hawasly, Tasnim Mohiuddin, Hamdy Mubarak and Sabri Boughorbel},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-main.36},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-main.36}
}

References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219. ↗
  2. Sultan Alrashed, Dmitrii Khizbullin, and David R Pugh. 2024. Fineweb-Edu-Ar: Machine-translated corpus to support Arabic small language models. arXiv preprint arXiv:2411.06402. ↗
  3. Tressy Arts, Yonatan Belinkov, Nizar Habash, Adam Kilgarriff, and Vit Suchomel. 2014. artenten: Arabic corpus and word sketches. Journal of King Saud University - Computer and Information Sciences, 26(4):357–371. Special Issue on Arabic NLP. ↗
  4. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The Belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
  5. Ali Elfilali, Hamza Alobeidli, Clémentine Fourrier, Basma El Amel Boussaha, Ruxandra Cojocaru, Nathan Habib, and Hakim Hacid. 2024. Open Arabic llm leaderboard. https://huggingface.co/spaces/OALL/ Open-Arabic-LLM-Leaderboard-v1. ↗
  6. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, and 23 others. 2025. Fanar: An arabic-centric multimodal generative ai platform. Preprint, arXiv:2501.13944. ↗
  7. May Farhat, Said Taghadouini, Oskar Hallström, and Sonja Hajri-Gabouj. 2024. ArabicWeb24: Creating a high quality arabic web-only pre-training dataset. find
  8. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, and 22 others. 2024a. OLMo: Accelerating the science of language models. Preprint. find
  9. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, and 1 others. 2024b. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838. ↗
  10. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations. ↗
  11. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024. AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics. ↗
  12. Fajri Koto, Haonan Li, Sara Shatanawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024.
  13. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, and 33 others. 2022. Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. Transactions of the Association for Computational Linguistics, 10:50–72.
  14. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. FineWeb-Edu: the finest collection of educational content. find
  15. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  16. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. In Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannheim. Leibniz-Institut für Deutsche Sprache. ↗
  17. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024a. The fineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, volume 37, pages 30811–30849. Curran Associates, Inc.
  18. Guilherme Penedo, Hynek Kydlíček, Alessandro Cappelli, Mario Sasko, and Thomas Wolf. 2024b. Datatrove: large scale data processing. find
  19. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  20. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, and 1 others. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. ↗
  21. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, and 17 others. 2024. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint. find
  22. Reem Suwaileh, Mucahid Kutlu, Nihal Fathima, Tamer Elsayed, and Matthew Lease. 2016. ArabicWeb16: A new crawl for today’s Arabic web. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’16, page 673–676, New York, NY, USA. Association for Computing Machinery. find
  23. Yudong Wang, Zixuan Fu, Jie Cai, Peijun Tang, Hongya Lyu, Yewei Fang, Zhi Zheng, Jie Zhou, Guoyang Zeng, Chaojun Xiao, and 1 others. 2025. Ultrafineweb: Efficient data filtering and verification for high-quality llm training data. arXiv preprint arXiv:2505.05427. ↗
  24. Yijiong Yu, Ziyun Dai, Zekun Wang, Wei Wang, Ran Chen, and Ji Pei. 2025. Opencsg chinese corpus: A series of high-quality chinese datasets for llm training. Preprint, arXiv:2501.08197. ↗
  25. Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, and 1 others. 2024. mGTE: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1393–1412.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hawasly-2025-arabicweb-edu-educational-quality-data/