📄 dzNLP at NADI 2024 Shared Task: Multi-Classifier Ensemble with Weighted Voting and TF-IDF Features

Summary

The dzNLP team took part in Subtask 1 of NADI 2024, Multi-label Country-level Dialect Identification (MLDID, Closed Track). In this task each Arabic sentence may be valid in several national dialects. The team chose classical machine learning over neural models. They trained on earlier NADI releases (2020, 2021, 2023) and tuned on a 100-sentence development set, then were scored on 1,000 test sentences. They ran three experiments:

• joining TF-IDF vectors from word, char and char_wb analyzers across n-gram ranges, with an SVC on top;
• the same three-way union with a separate weight for each analyzer, feeding a class-balanced LSVC;
• hard and weighted hard voting over LSVC, Random Forest and KNN.

The best development F1 was 22.51%, from the weighted feature union. The voting ensembles did worse than a carefully tuned single LSVC. On the official test set the system had the top precision among participants (63.22%), but recall was only 12.87%, so F1 was about 21%. The work shows that cheap sparse features can make very confident predictions while missing many valid labels.

Key points

• Task: NADI 2024 Subtask 1 (MLDID, Closed Track). The data covers dialects such as Egyptian, Saudi, Algerian, Syrian, Palestinian and Lebanese. The dev set has 100 samples and the test set 1,000. NADI 2020/2021/2023 data was allowed for training.

• Search space:

  • n-gram ranges (1,n) with n from 1 to 5;
  • TF-IDF transformer weights from 0.1 to 1;
  • max_features from 300 to 1000;
  • LSVC with C from 1 to 5 and class_weight='balanced';
  • RF with default settings;
  • KNN with n_neighbors=3;
  • voting weights from 0.1 to 0.6.

• Development F1 by setup:

  Setup	Dev F1
  Unigram TF-IDF + linear SVC (baseline)	19.43%
  Union of word/char/char_wb + balanced LSVC	20.64%
  Weighted union, best setting (n-grams (5,5,5), tw={0.65, 0.85, 0.85})	22.51%
  Plain hard voting	16.33%
  Weighted hard voting	21.44%

• Official test results: precision 63.22% (highest of all teams), recall 12.87%, F1 roughly 21%.

Interesting ideas

• I1. Giving each feature family (whole words, raw character spans, character spans kept inside word boundaries) its own tunable weight helped more than treating all families equally, which is a cheap way to let the data decide how much surface form versus lexical identity matters.
• I2. A properly tuned single model beat a voting ensemble of weaker, different classifiers, and plain majority voting was the worst setup of all. Combining models only pays off when each member is individually strong.
• I3. Setting class weights to 'balanced' gave a reliable gain when labels were unevenly distributed. This is a one-line counter to frequent classes dominating the decisions.
• I4. The high-precision, low-recall outcome shows that a single F1 number hides how a system fails. Reporting precision and recall separately reveals whether a model is cautious or noisy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): For out-of-vocabulary words, the tagger could score lemma or POS guesses by mixing suffix-character evidence and word-level evidence, with a weight tuned on held-out data per source.
• I2 (learn): When silver trees are accepted by agreement of several witness parsers, the agreed output should be benchmarked against the best single parser alone. This checks that adding a weak witness does not lower quality.
• I3 (tree): The perceptron parser's updates could be reweighted so that rare dependency relations (e.g., vocative, dislocated) are not swamped by nsubj/obj/det.
• I4 (answer): Readers answering school tests or fable QA should log abstentions and correct answers separately. A conservative reader that skips many questions then looks different from one that guesses wrongly.