📄 GHHT at CALCS 2018: Named Entity Recognition for Dialectal Arabic Using Neural Networks

Summary

The paper describes the GHHT team's entry to the CALCS 2018 shared task, which asked for named entity recognition on tweets that mix Modern Standard Arabic (MSA) and Egyptian Arabic (EGY). The data marks nine entity types with BIO tags but carries no code-switching labels, so the challenge is handling dialectal noise that trips up MSA-oriented tools. The authors use a bi-LSTM-CRF tagger. Each word is represented by a character-level CNN vector joined with a word2vec embedding, a Brown cluster feature and a gazetteer feature. The embeddings and clusters were trained on a web crawl of about 383M words mixing dialectal social media text and MSA news. The authors also repeat sentences that contain entities to counter class imbalance. The only preprocessing is conversion to Buckwalter transliteration, and no morphological analyzers, POS taggers or hand-written rules are used. Adding the components one at a time raises validation macro F from 66.49 to 77.97. On the test set the system scores 70.09 FB1 and places second, 1.5 points behind the winner. The work shows that cheap, optional resources can carry NER for a low-resource dialect.

Key points

• Data: The training set has 10,100 tweets (204,286 tokens) and the validation set 1,122 tweets (22,742 tokens). Only 11.3% of tokens are entities, with PERSON the most frequent (4.3%) and OTHER the rarest (0.06%).
• Architecture:
  • A char-CNN (window 4, 40 filters) feeds a bi-LSTM (state size 100), followed by a linear-chain CRF.
  • Training uses SGD with momentum 0.9, batch size 150, dropout 0.5 and early stopping with patience 35.
• External resources:
  • word2vec embeddings (300-dimensional) trained on the ~383M-word crawled corpus.
  • 100 Brown clusters trained on the same corpus.
  • A gazetteer of 40,719 unique names collected from Arabic Wikipedia and the Benajiba et al. corpus.
• Ablation on validation (macro F):
  • Baseline: 66.49
  • Adding characters: 69.60
  • Adding pre-trained embeddings: 72.38
  • Adding Brown clusters: 72.30 (no gain)
  • Adding oversampling: 73.05
  • Adding gazetteers: 77.97
  • Micro F stays near 96–97 throughout because the O label dominates.
• Test result: 70.09, against a shared-task baseline of 62.71 and a winning score of 71.62. Per-label accuracy is weakest for B-EVENT (43.48) and I-TIME (38.89).

Interesting ideas

• I1. Report macro F next to micro F in every ablation, since micro F hid almost all differences when "O" made up 88% of tokens.
• I2. Counter label imbalance by simply repeating (here 10×) every training sentence that contains a minority label, with no change to the model.
• I3. Feed a name-list lookup as a tiny extra input vector (dimension 2) rather than as a hard rule, so the model learns how much to trust it; this gave the largest single gain.
• I4. Brown clusters learned from a large unlabeled corpus give a cheap, discrete word-class signal, though here they did not help on top of embeddings, a useful negative result.
• I5. Build each resource as an optional plug-in layer so the system still runs, with lower quality, when a resource is missing for a language or domain.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluation of rare UD relations or rare frame roles should always include a macro, per-label view so that improvements on rare classes are not masked by frequent ones.
• I2 (learn): When building silver treebanks, sentences containing rare constructions (e.g. parataxis, vocative, rare VerbNet roles) could be duplicated before training the perceptron parser or gparse.
• I3 (world): Wikidata/WordNet name lookups could be supplied as soft features to entity detection, while the lookup hit is kept as an explainable trace of why an entity was proposed.
• I4 (tokens): Cluster IDs could be a transparent feature for the en tagger on unknown words, provided their benefit is checked by ablation rather than assumed.
• I5 (tree): gparse feature inputs (lexicons, clusters, gazetteers) could be toggled modules, which would make ablations trivial and allow porting to dialects with fewer resources.