📄 QC-GO Submission for MADAR Shared Task: Arabic Fine-Grained Dialect Identification

Summary

The paper reports the QC-GO team's entry in the MADAR Shared Task on fine-grained Arabic dialect identification. The task had two parts. Subtask 1 asks which of 25 city dialects, or MSA, a sentence from a parallel travel-domain corpus of about 110k sentences belongs to. Subtask 2 asks for a Twitter user's country, inferred from at most 100 of their tweets. The authors combined several classifiers.

• Neural model: a character-level CNN feeds into a highway layer and a multi-layer Bi-LSTM, together with word embeddings. Two variants were built, one using word2vec embeddings and one using fastText embeddings.
• FastText classifier: uses character n-grams.
• Gazetteer rule (Subtask 2 only): counts how often a user mentions Arab country and city names.

Outputs were merged by weighted or unweighted majority voting. The ensembles scored best on the development sets, with roughly 64–67% F1. On the test sets, however, a single component won: the FastText classifier in Subtask 1 and the place-name rule in Subtask 2 (67.3% F1). The work shows that noisy dialectal text with non-standard spelling calls for sub-word features. It also shows that cheap, transparent heuristics can rival neural models.

Key points

• The data come from the MADAR project. Subtask 1 uses travel-domain sentences in MSA paired with translations into 25 city dialects. Subtask 2 labels Twitter users by country, using up to 100 tweets each.
• The DNN builds word representations by applying a character CNN with max-over-time pooling and joining the result with 300-dimensional word embeddings. These embeddings were trained on the task's training set, giving the DNN-wv and DNN-ft variants. The combined vectors pass through a highway network, then a Bi-LSTM, then a softmax layer.
• The best FastText configuration used character n-grams of length 3 to 6, 100-dimensional vectors, a learning rate of 0.1 and 50 epochs.
• Subtask 1:
  • The best dev result was 64.53% F1, from the QC-GO1 ensemble.
  • The best test result was 60.42%, from FastText alone. The submitted ensembles reached about 57–59% on test.
• Subtask 2:
  • The best dev result was 66.60%, from the QC-GO2 ensemble.
  • The place-name heuristic gave the top test score, 67.30%. The neural models were far weaker here, with dev scores of about 44–47%.

Interesting ideas

• I1. A transparent rule outperformed learned models: the user is assigned the country whose names, or whose cities' names, appear most often in their tweets.
• I2. Building each word's vector from a character CNN plus a lookup embedding, passed through a highway layer, lets the model cope with spelling variants and unseen forms.
• I3. The ensemble weights each component by how well it did on the dev set, rather than treating all votes equally.
• I4. The ensembles won on dev but lost to a single component on test, which suggests the combination was overfitted to the tuning data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): Counting place-name mentions resolved against a Wikidata-backed gazetteer could set a default location for a text's world, with each mention kept as an inspectable justification.
• I2 (tokens): A similar character-aware component could make the tagger and lemmatizer more robust to typos and informal spellings in English input.
• I3 (learn): When several witness parsers vote on silver trees, their votes could be weighted by measured dev accuracy instead of simple agreement counts.
• I4 (learn): Every evaluation of combinations or repair rules should also be scored on data untouched during tuning, to catch overfitted gains.