📄 LDC Arabic Treebanks and Associated Corpora: Data Divisions Manual

Summary

LDC publishes its Arabic treebanks as single undivided collections, while NLP experiments need separate training, development and test portions. Without an agreed split, research groups either made their own divisions, which made results hard to compare, or reused older published ones that did not keep pace with new treebank releases. This technical report from Columbia's CCLS proposes one deterministic rule set for dividing these corpora. Documents are sorted alphabetically by name. DEV is filled from the top of the list and TEST from the bottom, each until it passes 10% of the corpus word count. Everything in between, about 80%, becomes TRAIN. The rule is applied to the Modern Standard Arabic treebanks ATB1–ATB12, the Egyptian treebanks ARZ1–ARZ8 and the spoken Levantine treebank, and the exact document ranges are listed for each. The report also explains how three earlier splits (Zitouni, MADA, JHU–Stanford) relate to the new 10-80-10 scheme. The value of the work is reproducibility: anyone can rebuild the split from a short description, and comparisons with earlier results stay valid.

Key points

• Design goals:
  • mandatory TRAIN/DEV/TEST sets;
  • DEV and TEST drawn from distant parts of the corpus;
  • splits made only at document boundaries;
  • no random selection;
  • each new TEST at least as large as the largest commonly used earlier TEST and containing its files.
• Projects that need extra subsets must take them from TRAIN, so DEV and TEST never change.
• ATB6 mixes newswire, newsgroup and weblog text, so each genre is treated as its own sub-treebank with its own three-way split.
• Zitouni's ATB3 split puts the first 509 documents (288,046 words, about 85%) in TRAIN and leaves a single DEVTEST. That DEVTEST includes 29 files that fall into 10-80-10 ATB3 TRAIN.
• The MADA ATB3 TEST set is 16 documents smaller than the 10-80-10 TEST, and the JHU–Stanford TEST set is a subset of the combined 10-80-10 TEST. Systems trained on the new TRAIN can therefore be scored on either older test set without contamination. The JHU DEVTEST, however, was chosen randomly and cannot be written down as a range.

Interesting ideas

• I1. Defining a split as contiguous ranges in a name-sorted file list makes the whole division describable in one table row and repeatable by anyone, with no random seed involved.
• I2. Placing DEV at one end of the sorted list and TEST at the other keeps the two sets far apart in the corpus, which lowers the chance that similar scores come from shared topics rather than real generalization.
• I3. Making the new TEST a superset of earlier test sets keeps old benchmarks usable, since dropping the extra documents recovers the earlier comparison.
• I4. Taking any extra subset (tuning, calibration) only out of TRAIN guarantees that DEV and TEST stay fixed across projects.
• I5. Splitting a multi-genre corpus separately for each genre stops one register from dominating or leaking into another's evaluation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): Treebank splits for the perceptron parser and gparse could be stored as first/last document IDs, so every evaluation run uses exactly the same held-out data.
• I2 (learn): When splitting silver data or tales for evaluation, held-out portions could be taken from distant ends of the source order to act as a check against topical leakage.
• I3 (answer): Each time the school-test or fable QA suite grows, earlier test items could be kept as a frozen core, so new scores stay comparable with past runs.
• I4 (learn): Data for self-induced repair rules or teacher-tier calibration could be cut from training material only, so the evaluation sets with negative controls remain untouched.
• I5 (tree): When mixing UD treebanks of different genres or dialect conversions, splitting each genre on its own would allow per-genre LAS reporting and avoid hidden cross-genre overlap.