Elamur Research⌕
☰
home / papers / 2019 / The Inception Team at NSURL-2019 Task 8: Semantic Question Similarity in Arabic

The Inception Team at NSURL-2019 Task 8: Semantic Question Similarity in Arabic

Paper · 2019 · Hana Al-Theiabat and Aisha Al-Sadi

Summary

The paper describes the winning system for NSURL-2019 Task 8, which asks whether two Arabic questions mean the same thing. This is a paraphrase-detection problem relevant to Q&A sites and search. The data is the Mawdoo3 question-pair corpus: short Arabic questions labelled as similar (1) or not (0). The only preprocessing was normalising different spellings of country names to one form.

The authors compared four neural approaches:

  • a CNN that encodes each question separately and scores the pair by cosine similarity;
  • a bidirectional GRU that reads both questions concatenated into one sequence;
  • an 8-head multi-head attention (transformer-style) network;
  • multilingual BERT fine-tuned as a sentence-pair classifier.

The non-pretrained models reached about 87–89% F1. Fine-tuned BERT reached about 96%. Combining several BERT runs trained with different random seeds by majority vote gave the official 95.924% private F1, which ranked first of nine teams. The work shows that a general multilingual pretrained model can dominate a modest-sized task in an under-resourced language.

Key points

  • Dataset: the Mawdoo3 dataset has 11,997 training pairs and 3,715 test pairs. About 55% of training pairs are labelled 0. Questions average roughly 5–6 words.
  • Baseline results (F1, public/private):
    • CNN: 88.330 / 88.773
    • BiGRU (128 units): 88.061 / 88.312
    • multi-head attention: 86.804 / 87.889
  • Multilingual BERT (max sequence length 50, learning rate 2e-5, batch size 8) scored 96.050 public and 95.617 private.
  • Ensembling and tie-breaking: the ensemble used hard voting over BERT runs with different seeds. When votes tied, the run with the best public score decided.
  • Submission choice: the 4-seed ensemble had the best public score (96.499), so it was submitted and scored 95.924 private. The 5-seed ensemble would actually have scored higher on the private split (96.232).

Interesting ideas

  • I1. Train the same model several times with different random seeds and take a majority vote, letting the most reliable member break ties; this is a cheap way to gain robustness without new architectures.
  • I2. Choosing the final model by the visible public split picked a weaker ensemble than one that was available, a concrete example of selection overfitting to a small evaluation slice.
  • I3. Normalising name variants of the same entity (country names) before comparison was the only preprocessing, yet it targets a real source of spurious mismatch between paraphrases.
  • I4. A cross-encoder that reads both questions jointly (BERT on the pair) clearly beat a model that encoded each question separately and then compared vectors.
  • I5. The paper stresses that paraphrased questions often differ mainly in word order, so a matcher should be insensitive to surface ordering.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when several witness parsers disagree on silver data, use majority agreement and break ties with the historically most accurate witness.
  • I2 (learn): keep a held-out set that is never used for model or rule selection, so reported gains are not artifacts of picking on the dev data.
  • I3 (world): map entity mentions to canonical IDs using Wikidata aliases before linking, so paraphrased questions point to the same entity.
  • I4 (answer): when checking whether two questions ask the same thing, compare their aligned frames and roles jointly rather than comparing independent summaries of each.
  • I5 (frames): judge question equivalence by who-did-what-to-whom frames extracted from the UD tree, which are naturally invariant to word reordering.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Jordan University of Science and TechnologyIrbid · Jordan15 works · 1,840 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Hana Al-Theiabat and Aisha Al-Sadi. 2019. The Inception Team at NSURL-2019 Task 8: Semantic Question Similarity in Arabic. Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{altheiabat2019inception,
  title = {{The Inception Team at NSURL-2019 Task 8: Semantic Question Similarity in Arabic}},
  author = {Hana Al-Theiabat and Aisha Al-Sadi},
  booktitle = {Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers},
  year = {2019},
  url = {https://aclanthology.org/2019.nsurl-1.17/}
}

References

14 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence similarity. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA., pages 2786–2792. find
  2. Georgios Paltoglou and Mike Thelwall. 2010. A study of information retrieval weighting schemes for sentiment analysis. In ACL 2010, Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, July 11-16, 2010, Uppsala, Sweden, pages 1386–1395. find
  3. Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA., pages 2793–2799. find
  4. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532–1543.
  5. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227–2237.
  6. Luca Ponzanelli, Andrea Mocci, and Michele Lanza. 2015. Summarizing complex development artifacts by mining heterogeneous data. In 12th IEEE/ACM Working Conference on Mining Software Repositories, MSR 2015, Florence, Italy, May 16-17, 2015, pages 401–405. ↗
  7. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI. find
  8. Haitham Seelawi, Ahmad Mustafa, Al-Bataineh Hesham, Wael Farhan, and Hussein T. Al-Natsheh. 2019. NSURL-2019 task 8: Semantic question similarity in arabic. In Proceedings of the first International Workshop on NLP Solutions for Under Resourced Languages, NSURL ’19, Trento, Italy.
  9. Aliaksei Severyn and Alessandro Moschitti. 2015. Learning to rank short text pairs with convolutional deep neural networks. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Santiago, Chile, August 9-13, 2015, pages 373–382. find
  10. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998–6008.
  11. Shuohang Wang and Jing Jiang. 2017. A compare-aggregate model for matching text sequences. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. ↗
  12. Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. arXiv preprint arXiv:1702.03814. ↗
  13. Zhiguo Wang, Haitao Mi, and Abraham Ittycheriah. 2016. Sentence similarity learning by lexical decomposition and composition. CoRR, abs/1602.07019. ↗
  14. Ho Chung Wu, Robert Wing Pong Luk, Kam-Fai Wong, and Kui-Lam Kwok. 2008. Interpreting TF-IDF term weights as making relevance decisions. ACM Trans. Inf. Syst., 26(3):13:1–13:37. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/theiabat-2019-inception-team-nsurl-2019-task/