Elamur Research⌕
☰
home / papers / 2025 / Wasm: A Pipeline for Constructing Structured Arabic Interleaved Multimodal Corpora

Wasm: A Pipeline for Constructing Structured Arabic Interleaved Multimodal Corpora

Summary

Arabic web corpora have generally been built as plain text. That approach discards page layout, formatting and the placement of images, which interleaved multimodal models need. Wasm addresses this gap with a Common Crawl processing pipeline for Arabic, adapted from the OBELICS framework. Instead of flat text, it outputs structured Markdown in which images stay where they appeared on the page, so the same data can serve text-only LLM pre-training or multimodal training.

The main adaptations are:

  • early Arabic language filtering that runs before WARC files are loaded,
  • KenLM perplexity filtering with thresholds tuned for Arabic and its dialects,
  • deduplication at the level of individual HTML nodes, using Needleman–Wunsch alignment, rather than whole documents.

The authors compare their filtering choices with FineWeb2, ArabicWeb24, CulturaX and the 101 Billion Arabic Words corpus. They release part of the pipeline code and a sample dataset dump, which was partly used to train the Baseer vision model.

This matters because it is presented as the first Arabic resource of its kind for interleaved, structure-preserving pre-training.

Key points

  • Wasm builds on OBELICS but outputs Markdown with interleaved images, stored in separate columns, instead of plain sequential text. This lets text and visual content be extracted separately.
  • Each Common Crawl dump is processed on its own. Pages with any Arabic content are kept even if not fully Arabic, and each page records its URL, source WARC file and byte offset.
  • Perplexity is the central quality criterion, using an Arabic KenLM model with dialect-aware thresholds. Under these thresholds, the share of excluded data is:
    • Wasm: 0%
    • FineWeb2: 1.766%
    • ArabicWeb24: 7.82%
    • CulturaX: 8.605%
    • 101 Billion Arabic Words: 19.757%
  • Near-duplicates are removed per HTML node using sequence alignment (Needleman–Wunsch). Repeated menus and boilerplate are dropped, while the unique parts of a page are kept.
  • Releases: code at github.com/misraj-ai/wasm and a dataset dump at huggingface.co/datasets/Misraj/msdd.

Interesting ideas

  • I1. Deduplicate at the level of document parts rather than whole documents, so a page with boilerplate still contributes its unique content instead of being discarded entirely. — For Mova (learn): Deduplicate the silver corpus by sentence or clause, so repeated textbook headers do not cost us the unique story sentences around them.
  • I2. Run the cheapest filter, language detection, before any expensive loading step; this saves time, memory and storage at scale. — For Mova (learn): Apply quick lexical and length checks to candidate sentences before sending them to several witness parsers or LLM teacher tiers.
  • I3. Calibrate a perplexity filter so it removes garbled text but still keeps legitimate variation such as dialects, rather than using thresholds tuned on a standard-language reference. — For Mova (learn): When filtering training text, tune thresholds on our own genres (fables, school tests, archaic tale language) so valid but unusual English is not filtered out.
  • I4. Audit your own quality filter by running it on competitors' corpora, reporting exclusion rates, and showing concrete rejected examples. — For Mova (learn): Run Mova's tree-sanity and repair checks on external treebanks and silver sets, and publish rejection rates with examples as an outside-data control.
  • I5. Keep a provenance pointer (source file plus byte offset) for every extracted item, so any datum can be traced back to its origin. — For Mova (world): Attach source offsets to every entity and event in the world of the text, so each answer can point to the exact span that supports it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2511.07080 · arXiv 2511.07080

@misc{chrouf2025wasm,
  title = {{Wasm: A Pipeline for Constructing Structured Arabic Interleaved Multimodal Corpora}},
  author = {Chrouf;, Khalil Hennara; Ahmad Bastati; Muhammad Hreden; Mohamed Motasim Hamed; Zeina Aldallal; Sara and AlModhayan, Safwan},
  year = {2025},
  doi = {10.48550/arxiv.2511.07080},
  eprint = {2511.07080},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2511.07080}
}

References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Manel Aloui, Hasna Chouikhi, Ghaith Chaabane, Haithem Kchaou, and Chehir Dhaouadi. 101 billion arabic words dataset, 2024. URL https://arxiv.org/abs/2405.01590. ↗ our card
  2. Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, and Muhammad Abdul-Mageed. Peacock: A family of Arabic multimodal large language models and benchmarks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl- long.689. URL https://aclanthology.org/2024.acl-long.689/. ↗
  3. Anas Awadalla, Le Xue, Oscar Lo, Manli Shu, Hannah Lee, Etash Guha, Sheng Shen, Mohamed Awadalla, Silvio Savarese, Caiming Xiong, et al. Mint-1t: Scaling open-source multimodal data by 10x: A multimodal dataset with one trillion tokens. Advances in Neural Information Processing Systems, 37:36805–36828, 2024. find
  4. Wei Chen, Lin Li, Yongqi Yang, Bin Wen, Fan Yang, Tingting Gao, Yu Wu, and Long Chen. Comm: A coherent interleaved image-text dataset for multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8073–8082, 2025. find
  5. Wasm (Ours) Impact / Motivation Maintains high quality while preserving Arabic linguistic structures Facilitates extraction of both text and visual content for multimodal model training Retains valid Arabic variation while filtering incoherent or low-quality text More accurate removal of near-duplicate Arabic content Supports training of multiple model types and tasks Balanced filtering adapted to Arabic Preserved structure with separate columns Central criterion with Arabic-tuned thresholds across dialects Sequence based alignment-Flexible for diverse data types and tasks Common Crawl Foundation. Statistics of common crawl monthly archives - distribution of languages, 2025. URL https://commoncrawl.github.io/cc-crawl-statistics/plots/languages.html. Accessed September 15, 2025. ↗
  6. May Farhat, Said Taghadouini, Oskar Hallström, and Sonja Hajri-Gabouj. Arabicweb24: Creating a high quality arabic web-only pre-training dataset, 2024. URL www.lighton.ai/lighton-blo gs/arabicweb24. find
  7. Matthieu Futeral, Armel Zebaze, Pedro Ortiz Suarez, Julien Abadji, Rémi Lacroix, Cordelia Schmid, Rachel Bawden, and Benoît Sagot. moscar: A large-scale multilingual and multimodal document-level corpus. arXiv preprint arXiv:2406.08707, 2024. ↗
  8. Kenneth Heafield. KenLM: Faster and smaller language model queries. In Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar F. Zaidan (eds.), Proceedings of the Sixth Workshop on Statistical Machine Translation, pp. 187–197, Edinburgh, Scotland, July 2011. Association for Computational Linguistics. URL https://aclanthology.org/W11-2123/. ↗ our card
  9. Khalil Hennara, Muhammad Hreden, Mohamed Motasim Hamed, Ahmad Bastati, Zeina Aldallal, Sara Chrouf, and Safwan AlModhayan. Baseer: A vision-language model for arabic document-to-markdown ocr. arXiv preprint arXiv:2509.18174, 2025. ↗ our card
  10. Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35:31809–31826, 2022. find
  11. Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023. URL https://arxiv.org/abs/2306.16527. ↗ our card
  12. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176bparameter open-access multilingual language model. 2023. find
  13. Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shenglong Ye, Zhenjiang Jin, Guanzhou Chen, Yinan He, Zhangwei Gao, Erfei Cui, et al. Omnicorpus: A unified multimodal corpus of 10 billion-level images interleaved with text. arXiv preprint arXiv:2406.08418, 2024. ↗
  14. Saul B. Needleman and Christian D. Wunsch. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 48(3): 443–453, 1970. ISSN 0022-2836. doi: https://doi.org/10.1016/0022-2836(70)90057-4. URL https://www.sciencedirect.com/science/article/pii/0022283670900574. ↗
  15. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages, 2023. URL https://arxiv.org/abs/2309.09400. ↗ our card
  16. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=n6 SCkn2QaG. ↗ our card
  17. Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. Fineweb2: One pipeline to scale them all–adapting pre-training data processing to every language. arXiv preprint arXiv:2506.20920, 2025. ↗
  18. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. ↗
  19. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. ↗ our card
  20. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278–25294, 2022. find our card
  21. Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal C4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06939, 2023. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hennara-2025-wasm-pipeline-constructing-structured-arabic/