Elamur Research⌕
☰
home / papers / 2019 / AtyNegar at NSURL-2019 Task 8: Semantic Question Similarity in Arabic

AtyNegar at NSURL-2019 Task 8: Semantic Question Similarity in Arabic

Paper · 2019 · Atieh Sharifi, Hossein Hassanpoor and Najmeh Zare Maduyieh

Summary

The paper describes the AtyNegar team's entry to NSURL-2019 Task 8, which asks whether two Arabic questions mean the same thing. The authors note that Arabic has complex morphology and few tools, so they use a classic supervised pipeline instead of an end-to-end neural model. Preprocessing strips diacritics, tatweel and extra spaces and fixes spacing around punctuation. Each question pair then gets 41 hand-built features in several families:

  • overlap of stemmed words, POS tags and named entities over 1–3-grams;
  • Levenshtein-based soft word matching and word-position alignment;
  • character-level common prefix, suffix, substring and subsequence;
  • IDF-weighted bag-of-words cosine;
  • FastText embeddings, Word Mover's Distance and doc2vec similarity.

The features are standardized, SelectKBest keeps the 38 most useful, and an SVM decides whether the pair is similar. On the official test set the system reached an F-measure of 82.58%, which beats the task baseline. The work shows that cheap, interpretable surface and embedding features remain competitive for a low-resource language. Its error analysis also shows where such features fail.

Key points

  • The system has 41 features in total and keeps 38 after SelectKBest. When k was tuned on the NSURL training data, k=38 gave F 0.808 (precision 0.834, recall 0.783), slightly above k=41 (0.806).
  • The three dropped features are precision, recall and F of NER 3-gram overlap. They never fire because the questions are too short to share three-token entity sequences.
  • In the feature ranking, unigram word-overlap precision is by far the strongest feature (score 4095). It is followed by unigram overlap F, the longest common suffix, higher-order overlap, WMD, Levenshtein features and doc2vec. Plain averaged FastText cosine ranks low.
  • The IDF weights, the FastText model (300-dimensional vectors) and the doc2vec model all come from Arabic Wikipedia. The classifier is SVC on zero-mean, unit-variance features.
  • Final test F-measure is 82.58%.
  • Error analysis shows two problems:
    • Removing stop words also deletes question words such as "who" and "when", which matter for similarity.
    • Extra non-essential information in one question misleads the system.

Interesting ideas

  • I1. Generic stop-word lists throw away interrogatives, yet these words carry the core of a question; the authors suggest treating question words as equivalence classes, for example "which year" ≈ "when".
  • I2. Ranking features revealed a family that can never fire on the target data, since short questions contain no shared NER 3-grams, which is a cheap way to find dead components.
  • I3. The authors give explicit fallback values for overlap precision, recall and F when counts are zero: 1 if both sides are empty, 0 if only one is, so degenerate pairs score consistently.
  • I4. An alignment feature records how far each word's best match is displaced in position (|i−j|), a soft signal that word order changed, which sometimes alters meaning and sometimes does not.
  • I5. As future work, the authors propose computing n-gram overlap over dependency-tree paths instead of surface sequences, and using semantic networks instead of edit distance for synonyms.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Question readers should map wh-phrases to a canonical asked-for slot (TIME, PERSON, PLACE) before matching, rather than discarding or literally comparing them.
  • I2 (learn): Evaluation could log how often each repair rule or feature actually triggers on a domain, so silently inert rules get flagged or pruned.
  • I3 (learn): Agreement metrics between witness parsers need similar documented conventions for empty sets, so silver-data filtering does not produce undefined or misleading scores.
  • I4 (frames): When comparing two frames, displacement between aligned role fillers could flag candidate argument swaps (who did what to whom) for explicit checking.
  • I5 (tree): Paraphrase or duplicate detection could compare overlapping UD head–dependent paths, with lemmas unified through WordNet synsets, keeping similarity explainable rather than bag-of-words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Atieh Sharifi, Hossein Hassanpoor and Najmeh Zare Maduyieh. 2019. AtyNegar at NSURL-2019 Task 8: Semantic Question Similarity in Arabic. Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{sharifi2019atynegar,
  title = {{AtyNegar at NSURL-2019 Task 8: Semantic Question Similarity in Arabic}},
  author = {Atieh Sharifi, Hossein Hassanpoor and Najmeh Zare Maduyieh},
  booktitle = {Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers},
  year = {2019},
  url = {https://aclanthology.org/2019.nsurl-1.5/}
}

References

13 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammad AL-Smadi, Zain Jaradat, Mahmoud AL-Ayy, and Yaser Jararweh. 2017. Paraphrase identification and semantic text similarity analysis in Arabic news tweets using lexical, syntactic, and semantic features. Information Processing and Management, 53(3):640-652. https://doi.org/10.1016/j.ipm.2017.01.002. ↗
  2. Fatma Elghannam. 2016. Automatic Measurement of Semantic Similarity among Arabic Short Texts. Communications on Applied Electronics (CAE), 6(2):16-21. https://doi.org/10.5120/cae2016652430. ↗
  3. Asli Eyecioglu and Bill Keller. 2016. ASOBEK at SemEval-2016 Task 1: Sentence Representation with Character N-gram Embeddings for Semantic Textual Similarity. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 736-740. find
  4. Wael Gomaa and Aly Fahmy. 2013. A survey of text similarity approaches. International Journal of Computer Applications, 68(13):13-18. https://doi.org/10.5120/11638-7118. ↗
  5. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning Word Vectors for 157 Languages. In Proceedings of the International Conference on Language Resources and Evaluation. Pages 3483-3487.
  6. Rafael Michael Karampatsis. 2015. CDTDS: Predicting Paraphrases in Twitter via Support Vector Regression. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 75–79. find
  7. Sarah Kohail, Amr Rekaby Salama, and Chris Biemann. 2017. STS-UHH at SemEval-2017 Task 1: Scoring Semantic Textual Similarity Using Supervised and Unsupervised Ensemble. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Pages 175-179. find
  8. El Moatez Billah Nagoudi and Didier Schwab. 2017. Semantic Similarity of Arabic Sentences withWord Embeddings. In proceedings of the Third Arabic Natural Language Processing Workshop (WANLP), pages 18-24. find
  9. Atish Pawar and Vijay Mago. 2018. Calculating the similarity between words and sentences using a lexical database and corpus statistics. arXiv:1802.05667. ↗
  10. Haitham Seelawi, Ahmad Mustafa, Hesham Al-Bataineh, Wael Farhan, and Hussein T.Al-Natsheh. 2019. {NSURL}-2019 Task 8: Semantic Question Similarity in Arabic. In Proceedings of the first International Workshop on NLP Solutions for Under Resourced Languages. Trento, Italy.
  11. Andrea Segura-Olivares, Alejandro Garc´ıa, and Hiram Calvo. 2013. Feature Analysis for Paraphrase Recognition and Textual Entailment. Research in Computing Science, 70:144-119. find
  12. Junfeng Tian, Zhiheng Zhou, Man Lan, and Yuanbin Wu. 2017. ECNU at SemEval-2017 Task 1: Leverage Kernel-based Traditional NLP features and Neural Networks to Build a Universal Model for Multilingual and Cross-lingual Semantic Textual Similarity. In Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 191–197. find
  13. Wafa Wali, Bilel Gargouri, and Abdelmajid Ben Hamadou. 2017. Enhancing the sentence similarity measure by semantic and syntactico-semantic knowledge. Vietnam Journal of Computer Science, 4(1):51-60. https://doi.org/10.1007/s40595-016-0080-2. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sharifi-2019-atynegar-nsurl-2019-task-8/