📄 Parameter and Data Efficient Continual Pre-training for Robustness to Dialectal Variance in Arabic

Summary

Arabic has many dialects, such as Egyptian, Levantine and Moroccan, that differ from Modern Standard Arabic (MSA) in vocabulary, morphology and even meaning. Dialectal text and dialect–English translations are scarce. The authors (Amazon Web Services) ask three things: whether continuing to pretrain multilingual BERT on Arabic is a cheaper but equally good alternative to building an Arabic model from scratch, and whether small dialect corpora or parallel corpora add further value. Because memory and latency were tightly limited, they use 4-layer BERT encoders. These are trained with MLM on roughly 326M deduplicated Arabic sentences, then further trained on 102K Egyptian Arabic sentences and on about 255K GALE/BOLT Arabic–English pairs with a TLM loss. On the eight ALUE tasks their models beat truncated AraBERT and mBERT. Continued mBERT matches the scratch model while needing about 60% of its training time. Dialect data clearly helps the scratch model but barely changes the mBERT-based one, and parallel data adds almost nothing. The findings are relevant wherever a standard variety dominates the data but people write in dialects.

Key points

• Evaluation uses the ALUE benchmark with eight tasks: FID, MDD, MQ2Q, SVREG, SEC, OOLD, OHSD and XNLI. Six of them contain dialectal data, mostly from Twitter. Baselines are AraBERT, AraBERT-Twitter and mBERT, each cut down to its first 4 layers.

• Three models are compared:

  • m-B-Ar: mBERT pretrained further for 800K steps.
  • B-Ar: trained from scratch for 1.3M steps.
  • t-B-Ar: like B-Ar, but with its own 64K WordPiece vocabulary.

  Average scores are 68.76, 68.99 and 71.91 respectively, against 65.50 for AraBERT.

• The pretraining corpus C1 combines Oscar, Arabic Wiki, CC100, Arabic Gigaword, Newswire, GALE and BOLT. Before training, sentences that consist mostly of repeated content are removed.

• Adding the Egyptian Oscar corpus (C2) raises B-Ar's average from 68.99 to about 73. The same step moves m-B-Ar only from 68.76 to 69.16.

• Adding the parallel data (C3) with TLM gives negligible or small gains. A sharp drop on FID cancels most of the improvements seen on the other dialectal tasks, so the authors do not claim a benefit.

Interesting ideas

• I1. The deduplication flags near-duplicates cheaply: hash every 10-token sliding window with MD5, count how often each hash appears across the corpus, and drop a sentence when 70% or more of its windows are repeated, while keeping one copy of exact duplicates. This catches boilerplate without pairwise comparisons. — For Mova (learn): Apply the same window-hash filter to silver corpora before witness parsers vote, so repeated templated sentences do not inflate agreement or skew training.
• I2. The authors keep only the bottom 4 layers of a pretrained 12-layer encoder and then fine-tune it, losing little accuracy while gaining speed and memory. This is a cheap way to get a small model that still carries pretrained knowledge. — For Mova (tree): gparse could start from a truncated pretrained encoder to cut GPU cost, after checking LAS against the full-depth version.
• I3. Training happens in ordered stages (broad corpus, then a small dialect corpus, then parallel pairs), and this worked better than mixing everything into one pool. A small in-domain stage at the end gives the scarce data a stronger voice. — For Mova (tree): After main treebank training, a short final pass on a small non-standard set such as tweets or informal dialogue could make the parser robust to that register without retraining from scratch.
• I4. The benchmark deliberately mixes standard and dialectal tasks and reports a separate average for the dialectal subset. This shows both the gain on new varieties and whether the standard variety regressed. — For Mova (learn): Score each adaptation against a standard-English control set alongside the target register, as a built-in negative control against hidden regressions.
• I5. The parallel-data result is examined task by task, which shows that one collapsing task (FID) hid gains elsewhere. An average alone would have suggested "no effect." — For Mova (answer): Break test results down per test type (school tests, maths problems, fable QA) so that one failing reader does not hide or fake progress in the others.