📄 Birzeit Arabic Dialect Identification System for the 2018 VarDial Challenge

Summary

This paper covers the Birzeit University entry to the Arabic Dialect Identification (ADI) task at VarDial 2018. The task is to label broadcast speech as Egyptian, Gulf, Levantine, North African or MSA. Organizers supplied 600-dimensional acoustic embeddings, Arabic ASR word transcripts, phone strings from four non-Arabic recognizers and the WAV audio. The test set was much larger than in 2017 and partly drawn from YouTube, so it differed from the training conditions.

The officially submitted run used majority voting over SVM and feed-forward networks trained on the embeddings. It placed third with 53.38%. After the deadline the authors built many more systems:

• GMM-UBM and GMM tokenizer models on MFCC features
• word and phoneme TF-IDF with SVM
• PRLM
• a Jaccard similarity-to-every-training-sentence representation

Compressing the embedding and similarity views through 30-neuron bottleneck layers reached 61.3% F1 macro. This exceeds every official score. An SVM fusion, trained with cross-validation on the test outputs, reached 68.77% F1 micro. The paper shows that cheap lexical views and acoustic views are strongly complementary.

Key points

• Data: 14,591 train, 1,566 dev and 6,837 test utterances. 5,435 test items were added to the original 1,492 from MGB-3, so the test set was about 4.5 times the 2017 size.

• Official result: the voting run of embedding-based SVM and DNN models ranked 3rd of six teams (0.5338). UnibucKernel was 1st (0.5892).

• Single-system scores (post-deadline):

  System	Score
  GMM-UBM (2048 mixtures)	35.1%
  GMM tokenizer (best)	47.86%
  Word TF-IDF (68,707-word vocabulary) + SVM	51.47%
  Jaccard sentence similarity (16,157 features) + SVM	51.89%
  Czech PRLM on dev (best)	38.9%
  Phoneme TF-IDF on dev	41.12%

• Bottleneck system: two 30-neuron bottleneck layers were concatenated into a 60-dimensional vector and fed to an SVM. It scored 61.3% F1 macro, about 2.5 points above the official winner.

• Fusion: SVM fusion of 5-score output vectors, using 7-fold cross-validation over the test data, gave 68.77% F1 micro. Because the fusion was fit on test-set outputs, this number is not directly comparable to the official results.

Interesting ideas

• I1. Represent each item by its overlap similarity to every labelled example and let a classifier learn from that profile; this exemplar view beat a much larger TF-IDF vector while needing no linguistic features.
• I2. Before fusing, check how often two systems are right on the same items versus only one of them (here 30% vs. 40%), since low overlap is what justifies combining them.
• I3. Squeeze two very high-dimensional, heterogeneous views through small bottleneck layers and concatenate the compact codes, so neither view drowns the other in the final classifier.
• I4. Turn a continuous signal into discrete symbols by taking the most likely mixture component per frame, then model the resulting token n-grams like phones.
• I5. Fitting fusion weights by cross-validation on the test outputs inflates results, and a new recording domain clearly lowered every system's score.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): we could profile new silver sentences by their Jaccard similarity to agreed training sentences, flagging ones with no close neighbours as out-of-distribution before trusting them.
• I2 (learn): compute this overlap between witness parsers per relation type to decide which witnesses actually add information to silver agreement.
• I3 (tree): gparse could merge lexical and contextual feature blocks via small learned bottlenecks, keeping each block's contribution inspectable as a separate short vector.
• I4 (tokens): continuous embedding states could be quantized into a small symbol inventory, giving the tagger discrete, countable morphology-like units it can explain.
• I5 (learn): keep combination weights tuned only on held-out dev data and add a deliberately out-of-domain negative-control set to expose such gains.