📄 Lahjati at NADI 2025 A ECAPA-WavLM Fusion with Multi-Stage Optimization

Summary

The paper describes Lahjati, a submission to the spoken Arabic Dialect Identification (ADI) subtask of NADI 2025. Each short 16 kHz audio clip must be assigned to one of eight dialects, namely Egyptian, Emirati, Jordanian, Moroccan, Algerian, Mauritanian, Yemeni or Palestinian. Labeled speech is scarce, and many clips come from noisy TV and film sources. The system runs two pretrained encoders side by side. ECAPA-TDNN, taken from SpeechBrain's VoxLingua107 model, supplies speaker and prosody cues. WavLM-base, mean-pooled over time, supplies contextual acoustic features. Each output is projected to 256 dimensions, the two are joined and normalized, and a small feedforward head makes the prediction. Training happens in stages: the encoders stay frozen at first, then the whole network is fine-tuned under a warmup, hold and cosine learning-rate schedule. On the validation leaderboard the system reached 94.08% accuracy and placed 3rd. On the blind test set it fell to roughly 51%. This large gap shows how fragile dialect ID can be under distribution shift, and it argues for evaluating on truly unseen data.

Key points

• Data: UBC-NLP/NADI2025_subtask1_SLID, with about 12,900 training and 12,700 validation clips (around 8 hours each). The UBC-NLP/NADI2025_subtask1_ADI_Test set has 6,268 samples. No external data and no augmentation were used.
• Architecture: 256-dim ECAPA-TDNN embeddings and 768-dim WavLM-base embeddings are each projected to 256 dims. They are concatenated into a 512-dim vector, passed through LayerNorm, and fed to a two-layer feedforward classifier with dropout.
• Optimization: AdamW with weight decay 1e-2 and gradient clipping at max norm 1.0. Encoders use a learning rate of 1e-5, while the projection and classifier layers use 1e-4. Batch size is 4.
• Schedule: Encoders are frozen for the first 8,000 steps. After that, all weights are trained with linear warmup, a constant hold, and cosine decay up to 100,000 steps. Training for 100k steps instead of 50k added about +2% validation accuracy.
• Results: Validation gave 94.08% accuracy with an average NIST LRE 2022 cost of 6.37%. The test set gave about 51.0% accuracy and about 49.0% cost. No ablation or error analysis was reported.

Interesting ideas

• I1. Two encoders that capture different aspects of the signal are mapped to equal-sized projections, then concatenated and normalized, so that neither one dominates the fused vector by sheer dimensionality.
• I2. New task-specific layers first learn on top of frozen pretrained encoders, and only afterwards is the whole network unfrozen, so random initial gradients cannot damage the pretrained weights.
• I3. Pretrained layers are given a learning rate ten times lower than the fresh head, which protects general knowledge while still letting the classifier adapt quickly.
• I4. Accuracy that looked excellent on validation collapsed on the blind test, which shows that a validation set drawn from the same pool can hide distribution shift.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could combine transformer token states with explicit features from the perceptron parser in the same way. Projecting both to a shared width and applying LayerNorm before the arc scorer would keep each source balanced.
• I2 (learn): when adapting gparse to a new treebank or to silver data, the encoder could stay frozen until the parser head stabilizes, and then joint fine-tuning could proceed with a low encoder learning rate.
• I3 (learn): any fine-tuning of the tagger or parser on small domain sets, such as fables or school tests, could use split learning rates to avoid catastrophic drift.
• I4 (learn): each evaluation should include a held-out, differently sourced test slice as a negative control. A large gap between in-pool and out-of-pool scores should be treated as a warning that the model is overfitting to its sources.