Elamur Research⌕
☰
home / papers / 2019 / ST NSURL 2019 Shared Task: Semantic Question Similarity in Arabic

ST NSURL 2019 Shared Task: Semantic Question Similarity in Arabic

Paper · 2019 · Mohamed Lichouri, Mourad Abbas, Besma Benaziz et al.

Summary

The paper describes an entry in NSURL 2019 Task 8, which asks whether two Arabic questions mean the same thing. The task is framed as binary classification, and the data is the Mawdoo3 question-pair set: 11,995 training pairs, with 3,715 test pairs listed in Table 1. The authors build three families of systems in parallel:

  • Lexical: string-distance measures from the textdistance package, each tuned with a threshold.
  • Statistical: scikit-learn linear classifiers over word, char and char_wb n-grams.
  • Neural: an off-the-shelf Siamese LSTM network.

They then test preprocessing variants on the best configurations: punctuation removal, stop-word filtering, normalization, ISRIStemmer stemming and NLTK POS tagging. The final system feeds LSVC with TF-IDF n-gram features concatenated with the lexical distance scores. It reached an average F1 of 83.13%, against a benchmark of about 71.7%, and ranked 7th of 9 teams. The work matters as a reminder that cheap, transparent features stay competitive on an under-resourced language, while a neural model that looked strong in development broke down on the hidden test data.

Key points

  • The best lexical measure was Smith-Waterman distance, with an F1 of 69.77% at a 30% threshold. Levenshtein (68.28%) and Needleman-Wunsch (67.93%) came next; Hamming and Mlipns performed poorly (under 48%).

  • Linear classifiers on word 4/5-grams (LSVC, SGD, PAG) scored roughly 73–75% F1. Bernoulli and Multinomial NB performed clearly worse.

  • The Siamese DNN reached about 79.9% F1 on development. Adding POS tags raised this to 80.55%, but that model collapsed to 59.97% (public) and 62.11% (private) on the test set.

  • The final model combines:

    • TF-IDF over word 5-grams, char 3-grams and char_wb 3-grams;
    • a vector of distance measures;
    • an LSVC classifier on top.

    It scored 83.57 (public) and 82.69 (private), compared with 71.99 and 71.43 for the benchmark.

  • Preprocessing gave modest gains: roughly +2 points for the lexical approach, +2 for the statistical one and +1 for the neural one.

Interesting ideas

  • I1. Turning similarity scores from many string-distance algorithms into a feature vector, and appending it to sparse n-gram features, lets one linear classifier weigh surface overlap and content cues together; it is a cheap hybrid that beat either part alone.
  • I2. Each distance measure was tuned by sweeping a decision threshold from 10% to 100% and keeping the best value, which gives every simple baseline a fair calibrated shot before comparing it with learned models.
  • I3. A neural model that improved on development data but fell by about 20 points on hidden test data shows why a held-out check must be run before committing to a configuration.
  • I4. Character n-grams restricted to word boundaries (char_wb) capture subword morphology without spanning across words, which helps in morphologically rich languages with spelling variation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): QA readers matching a question to stored answers could add alignment scores (e.g. Smith-Waterman over lemma sequences) as explicit, inspectable features next to frame-match evidence.
  • I2 (learn): evaluation of rule-based matchers could include a threshold sweep, so that heuristic controls are compared at their best and not at an arbitrary cutoff.
  • I3 (learn): when repair rules or parser variants are picked from dev gains, a separate untouched split should confirm them, flagging dev-overfit choices automatically.
  • I4 (tokens): char_wb-style features could back off the tagger or lemmatizer on unknown English words, giving morphology cues without crossing token boundaries.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Mohamed Lichouri, Mourad Abbas, Besma Benaziz and Abed Alhakim Freihat. 2019. ST NSURL 2019 Shared Task: Semantic Question Similarity in Arabic. Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lichouri2019st,
  title = {{ST NSURL 2019 Shared Task: Semantic Question Similarity in Arabic}},
  author = {Mohamed Lichouri, Mourad Abbas, Besma Benaziz and Abed Alhakim Freihat},
  booktitle = {Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers},
  year = {2019},
  url = {https://aclanthology.org/2019.nsurl-1.12/}
}

References

11 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lahsen Abouenour, Karim Bouzouba, and Paolo Rosso. 2010. An evaluated semantic query expansion and structure-based approach for enhancing arabic question/answering. International Journal on Information and Communication Technologies, 3(3):37–51. find
  2. Yassine Benajiba, Paolo Rosso, and Abdelouahid Ly-hyaoui. 2007. Implementation of the arabiqa question answering system’s components. In Proc. Workshop on Arabic Natural Language Processing, 2nd Information Communication Technologies Int. Symposium, ICTIS-2007, Fez, Morroco, April, pages 3–5. find
  3. W Brini, M Ellouze, and L Hadrich Belguith. 2009. Qasal: Un système de question-réponse dédié pour les questions factuelles en langue arabe. 9ème Journées Scientifiques des Jeunes Chercheurs en Génie Electrique et Informatique, Tunisia. find
  4. Davide Buscaldi, Paolo Rosso, José Manuel Gómez-Soriano, and Emilio Sanchis. 2010. Answering questions with an n-gram based passage retrieval engine. Journal of Intelligent Information Systems, 34(2):113–134. ↗
  5. Bassam Hammo, Hani Abu-Salem, and Steven Lytinen. 2002. Qarab: A question answering system to support the arabic language. In Proceedings of the ACL-02 workshop on Computational approaches to semitic languages, pages 1–11. Association for Computational Linguistics.
  6. Mohamed Lichouri, Mourad Abbas, Abed Alhakim Freihat, and Dhiya El Hak Megtouf. 2018. Word-level vs sentence-level language identification: Application to algerian and arabic dialects. Procedia Computer Science, 142:246–253. find
  7. FA Mohammed, Khaled Nasser, and HM Harb. 1993. A knowledge based arabic question answering system (aqas). ACM SIGART Bulletin, 4(4):21–30. ↗
  8. Preslav Nakov, Lluı́s Màrquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alhakim Freihat, Jim Glass, and Bilal Randeree. 2016. SemEval-2016 task 3: Community question answering. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California. Association for Computational Linguistics.
  9. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. Journal of machine learning research, 12(Oct):2825–2830. ↗
  10. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344.
  11. Haitham Seelawi, Ahmad Mustafa, Al-Bataineh Hesham, Wael Farhan, and Hussein T. Al-Natsheh. 2019. NSURL-2019 task 8: Semantic question similarity in arabic. In Proceedings of the first International Workshop on NLP Solutions for Under Resourced Languages, NSURL ’19, Trento, Italy.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lichouri-2019-st-nsurl-2019-shared-task/