📄 SENIT at AraFinNLP2024: trust your model or combine two

Summary

The paper describes the SENIT team's entry to Subtask 1 of the AraFinNLP 2024 shared task, which is multi-dialect intent detection for banking queries. The training data comes from ArBanking77, a translation of the English Banking77 into MSA and Palestinian Arabic, with 77 intent labels. The test set, however, also covers dialects that never appear in training. The authors fine-tuned several BERT-style encoders: CAMeLBERT, MARBERTv1 and MARBERTv2. They also built a two-encoder model that joins the [CLS] vectors of CAMeLBERT and a MARBERT variant and passes the result through a single linear classifier. Each model was trained for 3, 5 and 7 epochs with batch size 16. The best single model, MARBERTv2 at 5 epochs, also beat the fused model, so it became the final submission and placed sixth on the leaderboard. The main lesson is that simply stacking two encoders does not guarantee a gain. The authors suggest that attention-based weighting or a meta-model would be needed to get benefit from the combination.

Key points

• The Subtask 1 data has 24,018 labelled sentences: 21,554 for training and 2,464 for validation. The split between MSA and Palestinian is roughly even, and there are 77 intent classes.
• MARBERTv2 at 5 epochs scored the best results, with dev F1 0.8417 and test F1 0.8204. CAMeLBERT reached at most 0.8346 dev and 0.7744 test.
• The fused CAMeLBERT+MARBERT models came in lower, at 0.8106 and 0.8134 dev F1 and about 0.804–0.805 test F1.
• SENIT ranked 6th of 11 teams with micro-F1 0.82041, just behind fifth place (0.82083). The winning team scored 0.87731.
• In every model family, 5 epochs worked best, and training ran on a Kaggle Tesla P100.

Interesting ideas

• I1. When two encoders are fused by concatenating their [CLS] outputs into a flat linear layer, the result can be worse than the stronger encoder used alone, which shows that naive combination is a risky default.
• I2. The authors propose a learned gate, either attention over the combined representations or a separate meta-model on top of frozen feature extractors, that decides how much to trust each source.
• I3. The task trains on two language varieties and then tests on dialects that were never seen, which makes it an honest check of robustness to distribution shift.
• I4. A short sweep over epoch counts (3/5/7) showed one consistent sweet spot, but the gap between dev and test scores was still noticeable.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): When merging the verdicts of several witness parsers into silver data, always compare the merged result against the best single witness, and weight each witness rather than treating them all as equal.
• I2 (tree): gparse could use a small learned arbiter that chooses between the perceptron parser and the transformer parser for each arc, instead of using a fixed voting rule.
• I3 (tree): Keep a held-out treebank in a different dialect or genre after UD dialect conversion, so that parser accuracy is not measured only on in-distribution data.
• I4 (learn): Report test results next to dev results for every training setting, so that tuning to the dev set shows up as a measurable gap rather than being hidden.