📄 JHU System Description for the MADAR Arabic Dialect Identification Shared Task

Summary

The paper describes the Johns Hopkins entry to the MADAR shared task on fine-grained Arabic dialect identification. Subtask 1 assigns travel-domain sentences to one of 25 cities or MSA. Subtask 2 assigns Twitter users to their country. The main difficulties are very short inputs and the risk of overfitting to topic or proper names.

The team's core tool was Prediction by Partial Matching (PPM). This is a compression-derived, variable-order character language model with automatic backoff. One PPM model is trained per class, and the class that fits the text best is chosen. On top of this they built an ensemble of CNN, RNN and MLP submodels and tried fastText embeddings trained on extra Arabic data (ADC, Twitter LID, Reddit, Twitter 1% sample).

The ensemble won on Subtask 1 with a macro F1 of 63.4. A plain PPM model of order 3 was best on Subtask 2 with F1 54.3. Extra embedding data helped only marginally, and the ensemble barely outperformed its strongest member. The lesson is that a simple character LM is a very strong baseline. Diverse neural architectures may end up learning overlapping information.

Key points

• PPM order matters per task: on dev data, N=4 was best for Subtask 1 (0.591) and N=3 was best for Subtask 2 (0.543). Higher orders degraded noticeably on tweets (N=5 gave 0.287).
• Final scores: the Subtask 1 ensemble reached P 63.7 / R 63.4 / F1 63.4. Subtask 2 PPM-3 reached P 74.9 / R 46.5 / F1 54.3. For PPM-3, the input was normalized to Arabic characters only, and the user name was prepended.
• Ensemble vs. its parts (Subtask 1): CNN 0.545, RNN 0.554, MLP over PPM scores 0.591, full ensemble 0.634. On Subtask 2 the ensemble fell below PPM-3.
• Embeddings: 300-dim fastText cbow vectors (random, MADAR, MADAR+) gave 0.632/0.626/0.634 on Subtask 1 and 0.399/0.397/0.411 on Subtask 2. The small gain on Subtask 2 is attributed to the extra data being in-domain Twitter text.
• Practical issues:
  • 5–12% of Subtask 2 tweets were no longer retrievable.
  • The system was built to classify individual tweets, so user labels came from a majority vote over each user's tweets.
  • Errors mirrored known pairs such as TUN/SFX and BAS/BAG.
  • Subtask 2 confusions were notably one-directional (e.g., Iraq→Oman).

Interesting ideas

• I1. Classification by "which class model would compress this text best" with a PPM character model needs no feature engineering. It is very robust on short strings, and the backoff chain shows exactly which contexts drove the decision.
• I2. Each submodel is first trained alone to emit a hidden vector of a common size. It is then frozen, and the vectors are concatenated under a step-down stack of dense layers. This lets heterogeneous signals, including non-neural probability scores and metadata, be fused in a uniform way.
• I3. The authors treat "the ensemble barely beats its best member" as a diagnostic of redundant, correlated components rather than a mere disappointment. They suggest training later members on data reweighted by earlier members' predictions.
• I4. Lower-order context worked better on noisy, sparse tweets than on clean travel phrases, so model capacity was tuned per domain rather than globally.
• I5. Inspecting the direction of confusions, not just their counts, revealed lopsided error patterns that aggregate F1 hides.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): A per-class PPM could serve as a cheap, inspectable detector of input register or variety (e.g., British vs. American spelling, informal social-media text) before tagging, so the tagger can switch settings.
• I2 (learn): Witness parsers or LLM-teacher scores could be wrapped as frozen "distribution submodels" feeding a small arbiter, keeping each witness's contribution separable and auditable.
• I3 (learn): When silver data is agreed by several witness parsers, measure how often they err together and compare the agreement set against the single best parser. Doing this as a control checks that agreement is real evidence and not shared bias.
• I4 (tokens): Backoff or context-window settings for the morphology and lemma models could be chosen per text genre (school texts vs. user-generated text) on held-out data, instead of using one fixed value.
• I5 (answer): Evaluation of the readers could log asymmetric confusion between answer types or frame roles, exposing systematic one-way biases that call for targeted repair rules.