📄 QCRI@QALB-2015 Shared Task: Correction of Arabic Text for Native and Non-Native Speakers' Errors

Summary

The authors present the system QCRI entered in QALB-2015, a shared competition on automatically fixing errors in written Arabic. The task had two tracks: comments by native speakers on Aljazeera articles (L1) and essays by learners of Arabic (L2). The system does not use one general model. It runs an ordered cascade of handlers, each aimed at a known error type. These cover dialectal substitutions, letter normalization problems (alef forms, ta marbouta/ha, ya/alef maqsoura), elongated letters, wrongly split or merged words, and out-of-vocabulary words. A word bigram language model built from roughly ten years of Aljazeera news makes every contextual decision. The team also mined a correction dictionary from earlier QALB data. For L2 text they added candidate fixes for definiteness and gender-number agreement. Punctuation was mostly left alone because earlier attempts had given mixed results. The L1 run combined high precision with moderate recall, while the L2 runs scored much lower. This shows that learner errors, which often produce valid but wrong words, are harder for a context-light approach.

Key points

• The LM is a word bigram model trained on 234,638 Aljazeera articles. It was consulted before every correction step.
• The team extracted 221,460 errors from QALB-2014 and the QALB-2015 training split.
  • 10,255 of them became deterministic fixes: one observed correction, seen at least twice.
  • 2,076 words with a dominant (≥60%) correction were manually reviewed to build alternative lists.
  • 297 extra deterministic fixes were added for L2.
• A reverse look-up table with 905k normalized entries maps each normalized form to the surface spellings seen in the Aljazeera corpus. The LM then picks among them in context.
• There were three runs: QCRI-1-ALJ (L1), and QCRI-1-L2 and QCRI-2-L2 (L2).
  • L1 precision was about 84–85, with F1 in the 60s.
  • L2 F1 stayed around 25–28, with recall near or below 20.
  • On the QALB-2014 sets, the new system reached F1 64.17 (dev) and 63.81 (test).
• The authors' L2 error analysis found three main error families: definiteness in idafa constructs, gender-number agreement, and preposition choice. Learners also confused letters that sound alike or differ only in their dots.

Interesting ideas

• I1. The correction dictionary was mined with a consistency filter. Unambiguous, repeated fixes are applied without further checks, mostly-consistent ones get a manual review, and the rest are dropped. This keeps precision high without much labelling.
• I2. Instead of guessing how to restore a normalized word, the system indexes every attested spelling under its normalized key and lets context choose. Candidate generation stays restricted to forms that actually occur in the corpus.
• I3. Whether to split or merge a token is decided by comparing contextual LM scores of both versions. Candidate split points are limited to linguistically plausible positions, such as after non-connecting letters in words of five or more letters.
• I4. For learners' grammatical errors, the system generates morphological variants and lets the model pick the most probable one, without explicitly diagnosing the agreement error. The variants are with and without the article, and with and without the feminine marker.
• I5. The system is an ordered pipeline of narrow, named handlers, each tied to one error type. Every change can be attributed to a specific rule. The team also deliberately skipped a subtask (punctuation) where earlier evidence showed little gain.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Normalization of typos and informal spellings could be built the same way from logged corrections. Only high-agreement mappings would be auto-applied, and each one stays traceable to its evidence count.
• I2 (tokens): The lemmatizer or spelling restorer could produce candidates only from attested surface forms per normalized key. The choice would then be explainable as "these forms exist, this one fits the context."
• I3 (tokens): Hyphenation, compounds and glued words (e.g., "alot", "can not") could be handled by scoring both segmentations in context. Splits would only be tried at a small set of rule-licensed points.
• I4 (back): After generation, the system could enumerate agreement and determiner variants of each noun phrase and select by score. This would serve as a cheap, explainable agreement check before text is emitted.
• I5 (learn): Self-induced repair rules could be organized as an ordered, per-type cascade with logged firings. Rule families could be kept or dropped based on measured gains against held-out data.