📄 dzFinNlp at AraFinNLP: Improving Intent Detection in Financial Conversational Agents

Summary

The dzFinNlp team tackled Subtask-1 of the AraFinNLP shared task: assigning each Arabic banking customer query to one of 77 intents. The data is ArBanking77, which was built by translating the English Banking77 corpus into MSA and Palestinian Arabic and was then extended to more dialects. The authors trained only on the Palestinian (PAL) portion. They then evaluated on a test set that mixes MSA, Gulf, Levantine and North African queries. They compared four system families:

• LinearSVC on combined TF-IDF n-gram features, unweighted;
• the same model with per-feature-block weights;
• LSTM and BiLSTM networks;
• frozen multilingual Sentence Transformer embeddings fed to logistic regression.

The tuned TF-IDF + LinearSVC system was the clear winner on the development set, at roughly 93% micro F1. On the multi-dialect test set it fell to 67.21%. The work shows that a carefully tuned sparse baseline can beat untuned neural models on short, domain-specific queries. It also shows that skill learned on a single dialect transfers poorly to others.

Key points

• Subtask data: 10,821 training, 1,234 development and 1,721 test sentences (13,776 in total). Queries average about 8 words and about 42–43 characters. The full ArBanking77 resource contains 31,404 queries across 77 intent classes.
• Features were built with scikit-learn's FeatureUnion over three TF-IDF analyzers: word, char and char_wb.
  • The block weights (transformer_weights) were searched from 0.1 to 1.0 in steps of 0.1.
  • The best settings paired word 3-grams or 4-grams with character 4–5-grams.
  • The strongest configurations set C between 4 and 6.
• Development F1 rose from 88.01 with a word-unigram baseline to between 92.11 and 93.08 with richer n-grams and weighting. The abstract reports 93.02 as the best figure, while the table's top value is 93.08.
• Neural results on development were much lower:
  • The plain LSTM (100 units, embedding size 100) scored 75.23.
  • Capping sequence length at 50 and vocabulary at 5,000 raised this to 79.6 for the LSTM and 79.84 for the BiLSTM.
  • Both xlm-r Sentence Transformer variants with logistic regression scored 75.76.
• The best system scored 67.21 micro F1 on the official multi-dialect test set. The gap from development reflects training on PAL only.

Interesting ideas

• I1. Combining word, character and word-boundary-aware character n-gram views, each with its own tunable weight, lets one linear model capture both lexical cues and sub-word morphology in noisy text.
• I2. The authors trained on one dialect and tested on a mixture of dialects, which exposed a drop of about 26 points that an in-dialect dev set completely hid.
• I3. A tuned sparse linear baseline beat LSTMs and frozen multilingual embeddings by more than 13 points, so cheap baselines deserve real tuning before neural models are credited with gains.
• I4. Shrinking the LSTM's input, through shorter maximum sequences and a capped vocabulary, improved scores by about 4–5 points, which suggests that pruning rare items can act as useful regularization on short-query tasks.
• I5. The 77 banking intents work as a fixed inventory of request types attached to whole utterances, essentially domain-specific speech acts.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): A weighted char/char_wb view could back an explainable guesser for the POS and lemma of out-of-vocabulary or misspelled words, with each view's contribution inspectable.
• I2 (learn): Every evaluation should include a deliberately shifted split, such as a different genre or register, so that in-domain scores are never read as general competence.
• I3 (learn): Keeping a well-tuned simple witness alongside the parsers and LLM teachers gives an honest floor for judging whether the complex components add value.
• I4 (tree): Capping rare word forms in the gparse or perceptron feature sets could be tested as a simple regularizer and checked on held-out treebanks.
• I5 (frames): A closed intent inventory like Banking77's could serve as a test bed for mapping prag's speech-act and frame output onto concrete request types, with each decision traced back to tree evidence.