Elamur Research⌕
☰
home / papers / 2019 / NSURL-2019 Shared Task 8: Semantic Question Similarity in Arabic

NSURL-2019 Shared Task 8: Semantic Question Similarity in Arabic

Paper · 2019 · Haitham Seelawi, Ahmad Mustafa, Hesham Al-Bataineh et al.

Summary

This paper reports on Shared Task 8 of the NSURL 2019 workshop on under-resourced languages. Systems had to decide whether two Arabic questions mean the same thing, even when worded differently. The organizers define two questions as equivalent if they share the same answer; a partly different answer counts as non-equivalent. Arabic is morphologically rich and has few labeled question-pair corpora, so Mawdoo3 built and released a new resource: 15,712 question pairs labeled by in-house annotators. The pairs are balanced between similar and dissimilar, and between factoid and non-factoid questions. The competition ran on Kaggle with a reproducible baseline and drew 9 teams. Four teams described their systems:

  • a fine-tuned multilingual BERT ensemble,
  • an ELMo-based siamese network trained on augmented data,
  • a stacked BiGRU over AraVec embeddings,
  • a classical n-gram pipeline.

Contextual subword or character models came out on top, with BERT reaching about 96% accuracy. The authors attribute this to Arabic morphology: a model that learns one vector per surface form, like Word2Vec, cannot share knowledge across the many variants of a root. The public dataset gives Arabic Q2Q research a benchmark.

Key points

  • The Mawdoo3 Q2Q dataset has 15,712 pairs with binary labels.
    • Train: 11,997 pairs. Test: 3,715 pairs.
    • Overall: 7,115 similar and 8,597 not similar.
    • Questions are at most 15 words long, and both sides of a pair have similar length distributions.
  • Participation: 9 teams made 547 Kaggle submissions in total, averaging over 60 per team. Four teams submitted system descriptions.
  • Winner: The Inception scored 0.95924 accuracy by fine-tuning multilingual BERT. They reran their best configuration with different random seeds and combined the runs by voting.
    • A 3-model ensemble did best on the public split.
    • Ensembles of 4–6 models did best on the private split.
  • Runners-up:
    • Tha3aroon (0.94848) fed ELMo embeddings into weight-shared ON-LSTM layers with sequence weighted attention. They merged the two question vectors with a pairwise squared distance and trained on a dataset augmented to 45,514 samples.
    • onekaggler (0.94809) stacked three BiGRU layers on AraVec tweets_cbow_300 embeddings, scored pairs with a dot product, and trained with MSE loss.
  • The n-gram approach lagged well behind. Speech Translation combined word, char and char_wb n-grams (1–5) with stemming and POS tags and found LSVM their best classifier, finishing 7th with 0.82698.

Interesting ideas

  • I1. Judging two questions equivalent only when their answers coincide gives a concrete test, instead of a vague notion of "similar wording."
  • I2. On a small dataset, the best submission did not search for more model variety; it repeated one good configuration across several random seeds and let the runs vote. The ideal ensemble size differed between the public and private splits.
  • I3. The organizers explain BERT and ELMo's lead by their ability to share learning across the many inflected forms of one root, which per-form embeddings cannot do.
  • I4. Several strong systems encoded both questions with one shared-weight encoder and compared them through a simple distance, such as squared difference or dot product. This keeps the comparison symmetric and lets each question's representation be computed once and reused.
  • I5. onekaggler scored 99% on its own validation set but under 94% on the test set, a clear sign that an internal split can mislead.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-test and fable QA readers could merge two question phrasings only when both resolve to the same node in the text's world, giving a checkable criterion for paraphrase.
  • I2 (learn): gparse could be retrained under several seeds and the runs combined by voting. Seed-to-seed disagreement would then mark unstable arcs, and ensemble size should be chosen on held-out data, not on the dev set.
  • I3 (tokens): This supports keeping lemma and morphological features explicit in the en crate, so that rare inflections inherit evidence from their lemma instead of being treated as unseen words.
  • I4 (world): Frames or entity descriptions could be encoded once and compared by a fixed, inspectable distance when checking coreference candidates or duplicate facts.
  • I5 (learn): Evaluation should always include a truly held-out set, and the gap between validation and test should be tracked, so that silver-data gains are not overestimated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Haitham Seelawi, Ahmad Mustafa, Hesham Al-Bataineh, Wael Farhan and Hussein T. Al-Natsheh. 2019. NSURL-2019 Shared Task 8: Semantic Question Similarity in Arabic. arXiv:1909.09691.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1909.09691 · arXiv 1909.09691

@misc{seelawi2019nsurl,
  title = {{NSURL-2019 Shared Task 8: Semantic Question Similarity in Arabic}},
  author = {Haitham Seelawi, Ahmad Mustafa, Hesham Al-Bataineh, Wael Farhan and Hussein T. Al-Natsheh},
  year = {2019},
  doi = {10.48550/arxiv.1909.09691},
  eprint = {1909.09691},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.09691}
}

References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Radu Soricut and Eric Brill. Automatic question answering: Beyond the factoid. In Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2004, Boston, Massachusetts, USA, May 2-7, 2004, pages 57–64, 2004. find
  2. Daniel M. Cer, Mona T. Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity - multilingual and cross-lingual focused evaluation. CoRR, abs/1708.00055, 2017. ↗
  3. Marwah Alian and Arafat Awajan. Arabic semantic similarity approaches–review. In The 19th International Arab Conference on Information Technology (ACIT’ 2018), 2018. find
  4. Goutam Majumder, Partha Pakray, Alexander F. Gelbukh, and David Pinto. Semantic textual similarity methods, tools, and applications: A survey. Computación y Sistemas, 20(4), 2016. ↗
  5. George A. Miller and Christiane Fellbaum. Wordnet then and now. Language Resources and Evaluation, 41(2):209–214, 2007. ↗
  6. Frane Saric, Goran Glavas, Mladen Karan, Jan Snajder, and Bojana Dalbelo Basic. Takelab: Systems for measuring semantic text similarity. In Proceedings of the 6th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2012, Montréal, Canada, June 7-8, 2012, pages 441–448, 2012.
  7. Gizem Sogancioglu, Hakime Öztürk, and Arzucan Özgür. BIOSSES: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics, 33(14):i49–i58, 2017. find
  8. Mohammad Taher Pilehvar, David Jurgens, and Roberto Navigli. Align, disambiguate and walk: A unified approach for measuring semantic similarity. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers, pages 1341–1351, 2013. find
  9. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States., pages 3111–3119, 2013. ai.mawdoo3.com ↗
  10. Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  11. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018. ↗
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. ↗
  13. John Wieting and Douwe Kiela. No training required: Exploring random encoders for sentence classification. CoRR, abs/1901.10444, 2019. ↗
  14. Jonas Mueller and Aditya Thyagarajan. Siamese recurrent architectures for learning sentence similarity. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA., pages 2786–2792, 2016. find
  15. Paul Neculoiu, Maarten Versteegh, and Mihai Rotaru. Learning text similarity with siamese recurrent networks. In Proceedings of the 1st Workshop on Representation Learning for NLP, Rep4NLP@ACL 2016, Berlin, Germany, August 11, 2016, pages 148–157, 2016. find
  16. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 3294–3302, 2015. ↗
  17. Chris Olah and Shan Carter. Attention and augmented recurrent neural networks. Distill, 2016. ↗
  18. Zhouhan Lin, Minwei Feng, Cícero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. CoRR, abs/1703.03130, 2017. ↗
  19. Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Learning semantic textual similarity from conversations. In Proceedings of The Third Workshop on Representation Learning for NLP, Rep4NLP@ACL 2018, Melbourne, Australia, July 20, 2018, pages 164–174, 2018. ↗
  20. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Universal sentence encoder. CoRR, abs/1803.11175, 2018. ↗
  21. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.05250, 2016. ↗
  22. Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge. CoRR, abs/1808.07042, 2018. ↗
  23. Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. arXiv preprint arXiv:1810.09536, 2018. ↗
  24. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. arXiv preprint arXiv:1708.00524, 2017. ↗
  25. Abu Bakr Soliman, Kareem Eissa, and Samhaa R. El-Beltagy. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256 – 265, 2017. Arabic Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/seelawi-2019-nsurl-2019-shared-task-8/