📄 AMCrawl: An Arabic Web-Scale Dataset of Interleaved Image-Text Documents and Image-Text Pairs

Summary

Most vision-language pretraining relies on image-text pairs scraped from alt-text. These captions are often empty, are just file names, or are short and unrelated to the image. Interleaved documents, where images sit inside running text, give much richer context, but public resources of this kind (MMC4, OBELICS, MINT-1T) are essentially English-only.

The authors adapt the OBELICS pipeline to Arabic and run it over a CommonCrawl snapshot. They filter pages for quality and safety and keep each image in its place within the document layout. They then prompt GPT to turn the documents into question-answer pairs tied to images, and they also produce Arabic translations of several standard multimodal datasets.

To test the resource, they train a 13B-parameter Arabic VLM in two stages and score its answers with GPT-4o as a judge. Fine-tuning on AMCrawl-QA clearly beats fine-tuning only on translated English data. The released data and code give Arabic multimodal research a native, culturally grounded foundation instead of relying solely on translation.

Key points

• Scale: 8.6M Arabic web pages were processed, giving 5.8M images and 1.3B text tokens. The final release has 2.8M interleaved documents and 5M QA pairs (AMCrawl-QA).
• Training recipe:
  • Stage 1 aligns only the projection layer on Arabic-translated CC3M-595K and MSCOCO (about 1.5M samples). Loss fell from 7.54 to 1.74.
  • Stage 2 tunes the projection layer and the LLM on AMCrawl QA, with the vision encoder frozen. Loss fell from 1.77 to 0.08.
• Evaluation follows the LLaVA protocol on 45k sampled test QA pairs, with GPT-4o as judge. The mean score rose from 32.3 (MIMIC+LLaVA data) to 43.6 after AMCrawl fine-tuning.
• Safety filtering had two steps:
  • URLs were screened for explicit keywords before download.
  • Downloaded images were checked by a MobileNet-based NSFW classifier, and any document with a flagged image was dropped. This removed 44,701 documents in total.
• Stated limitations:
  • The data comes from a single snapshot (June 2024).
  • Perceptual hashing suggests only about 45% of images are unique.
  • No human evaluation and no standard benchmarks yet; Henna and CVQA testing is ongoing.

Interesting ideas

• I1. Instead of trusting a short attached label, the pipeline uses the page's DOM structure to keep each item embedded in its real surrounding context, which carries far more meaning than a caption.
• I2. Grounded QA pairs are generated automatically by an LLM from real documents, turning unlabeled web pages into instruction data at scale.
• I3. The unit of rejection is the whole document: one flagged component discards everything around it, trading some yield for cleaner data.
• I4. Training proceeds in stages: first only a small bridging layer is fitted while the big components stay frozen, then more of the model is unfrozen.
• I5. The authors openly measured near-duplicates with perceptual hashing and reported low uniqueness rather than hiding it.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): when collecting raw text for coreference and role training, keep whole documents with their paragraph structure rather than isolated sentences, so pronouns and event chains have their true antecedents.
• I2 (answer): have an LLM teacher draft questions over fables and tales, then keep only those whose answers Mova's world-of-the-text can actually derive, yielding explainable training and test items.
• I3 (learn): if any sentence in a document fails witness-parser agreement badly, drop the full document from silver data so context-dependent errors do not leak in.
• I4 (tree): when attaching a new head or a new UD dialect to gparse, first train only the head on a frozen encoder, then fine-tune jointly, which stabilizes early training.
• I5 (learn): hash normalized sentences across treebanks and silver corpora to quantify duplication and to guarantee that no evaluation sentence leaks into training.