Elamur Research⌕
☰
home / papers / 2019 / Tha3aroon at NSURL-2019 Task 8: Semantic Question Similarity in Arabic

Tha3aroon at NSURL-2019 Task 8: Semantic Question Similarity in Arabic

Paper · 2019 · Ali Fadel, Ibraheem Tuffaha and Mahmoud Al-Ayyoub

Summary

The paper describes the winning-tier system of team Tha3aroon for NSURL-2019 Task 8, Semantic Question Similarity (SQS) in Arabic. The task is to decide whether two Arabic questions mean the same thing. The data is a roughly 12K-pair dataset from Mawdoo3, hosted on Kaggle.

The authors make two main contributions:

  • Data augmentation from logical properties. They treat similarity as an equivalence relation and use transitivity, symmetry and reflexivity to expand the training set to about four times its original size. This also balances the two classes.
  • A Siamese-style neural model.
    • Input: contextual embeddings from Arabic ELMo.
    • Encoder: two bidirectional ON-LSTM layers followed by sequence-weighted self-attention, with shared weights for both questions.
    • Merging: the element-wise squared difference of the two question vectors.
    • Classifier: a stack of dense layers.

The system reached 96.499 F1 on the public leaderboard, tied for first. On the private leaderboard it scored 94.848 F1 and placed second.

Ablations show three things:

  • ON-LSTM beats GRU and LSTM.
  • The full augmentation pipeline gives the best scores.
  • Replacing ELMo with FastText, or machine-translating the data into English, hurts results.

The work shows that cheap, rule-derived extra data can significantly help a low-resource pair-classification task.

Key points

  • The original training set has 11,997 pairs: 5,397 similar and 6,600 not similar. Augmentation grows it step by step:
    • transitive rules: 17,487 pairs;
    • symmetric rule: 34,974 pairs;
    • reflexive rule: 45,514 pairs (23,082 positive, 22,432 negative).
  • Model settings:
    • Arabic ELMo (ELMoForManyLangs) produces 1024-dim embeddings.
    • Each of the two BiLSTM-style ON-LSTM layers has 256 units, chunk size 8 and 20% dropout.
    • Attention pooling produces a 512-dim question vector.
    • The head has four ReLU dense layers (1024/512/256/128) and a sigmoid output.
  • Training setup: Adam, learning rate 0.001, binary cross entropy, batch size 256, 100 epochs, on a Colab Tesla T4.
  • Each configuration was trained five times. The authors report min, max and average F1, plus a majority-vote ensemble. The best vote scores were 96.499 (public) and 94.848 (private).
  • Negative results:
    • FastText input: 94.254 / 93.118 F1.
    • Translating to English with Google Translate: 88.868 / 87.504 F1.
    • Using the final LSTM state instead of attention pooling: about 1 F1 point lower.
  • Augmentation steps are not uniformly helpful. Adding only the transitive examples slightly lowered F1 compared with the original data, and only the full set gave the top score.

Interesting ideas

  • I1. Labels on pairs that form an equivalence relation can be closed under transitivity, symmetry and reflexivity to produce new labelled examples for free. Mixing a positive link with a negative one yields extra negatives as well.
  • I2. Each augmentation step was ablated separately, which revealed that one step on its own actually hurt. This is a reminder that "more derived data" needs a per-step check rather than blind trust.
  • I3. Reporting min, max, mean and majority vote over five training runs separates real gains from seed luck.
  • I4. Translating the input into a better-resourced language and solving there lost a lot of accuracy, which the authors attribute to information lost in translation.
  • I5. The attention weights picked out the words that define what a question asks about, such as the head noun versus a modifier like "most beautiful said about". This is how the model told apart questions that share words but differ in intent.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): silver paraphrase or coreference pairs agreed by witness parsers could be expanded by equivalence closure, with the derived pairs tagged so their provenance stays explainable.
  • I2 (learn): when adding self-induced repair rules or generated silver data, evaluate each batch of rules separately against a held-out set before merging.
  • I3 (learn): the gparse or perceptron parser comparisons could report spread across several seeds plus an ensemble vote, so LAS differences are not over-read.
  • I4 (back): translation through the meta-world should be checked with a round-trip task score, because a pivot that drops meaning can silently damage downstream decisions.
  • I5 (answer): question matching in QA could compare the wh-target and its modifiers in the UD tree explicitly, giving a structural and explainable version of this focus-word signal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ali Fadel, Ibraheem Tuffaha and Mahmoud Al-Ayyoub. 2019. Tha3aroon at NSURL-2019 Task 8: Semantic Question Similarity in Arabic. Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{fadel2019tha3aroon,
  title = {{Tha3aroon at NSURL-2019 Task 8: Semantic Question Similarity in Arabic}},
  author = {Ali Fadel, Ibraheem Tuffaha and Mahmoud Al-Ayyoub},
  booktitle = {Proceedings of the First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019 - Short Papers},
  year = {2019},
  url = {https://aclanthology.org/2019.nsurl-1.8/}
}

References

22 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammad AL-Smadi, Zain Jaradat, Mahmoud Al-Ayyoub, and Yaser Jararweh. 2017. Paraphrase identification and semantic text similarity analysis in arabic news tweets using lexical, syntactic, and semantic features. Information Processing & Management, 53(3):640–652. ↗
  2. Marwah Alian and Arafat Awajan. 2018. Arabic semantic similarity approaches-review. In 2018 International Arab Conference on Information Technology (ACIT), pages 1–6. IEEE. find
  3. Jianpeng Cheng, Zhongyuan Wang, Ji-Rong Wen, Jun Yan, and Zheng Chen. 2015. Contextual text understanding in distributional semantic space. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 133–142. ACM. find
  4. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. ↗
  5. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. ↗
  6. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. arXiv preprint arXiv:1708.00524. ↗
  7. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗
  8. Hua He, Kevin Gimpel, and Jimmy Lin. 2015. Multi-perspective sentence similarity modeling with convolutional neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1576–1586.
  9. Tiago Carneiro, Raul Victor Medeiros Da Nóbrega, Thiago Nepomuceno, Gui-Bin Bian, Victor Hugo C De Albuquerque, and Pedro Pedrosa Reboucas Filho. 2018. Performance analysis of google colaboratory as a tool for accelerating deep learning applications. IEEE Access, 6:61677–61685. ↗
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  11. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605. find
  12. Nitin Madnani, Joel Tetreault, and Martin Chodorow. 2012. Re-examining machine translation metrics for paraphrase identification. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 182–190. Association for Computational Linguistics.
  13. Adnen Mahmoud, Ahmed Zrigui, and Mounir Zrigui. 2017. A text semantic similarity approach for arabic paraphrase detection. In International Conference on Computational Linguistics and Intelligent Text Processing, pages 338–349. Springer. find
  14. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. ↗
  15. Didier Schwab et al. 2017. Semantic similarity of arabic sentences with word embeddings.
  16. Haitham Seelawi, Ahmad Mustafa, Hesham Al-Bataineh, Wael Farhan, and Hussein T. Al-Natsheh. 2019. NSURL-2019 task 8: Semantic question similarity in arabic. In Proceedings of the first International Workshop on NLP Solutions for Under Resourced Languages, NSURL ’19, Trento, Italy.
  17. Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2018. Ordered neurons: Integrating tree structures into recurrent neural networks. arXiv preprint arXiv:1810.09536. ↗
  18. Noah A Smith. 2019. Contextual word representations: A contextual introduction. arXiv preprint arXiv:1902.06006. ↗
  19. Chuanqi Tan, Furu Wei, Wenhui Wang, Weifeng Lv, and Ming Zhou. 2018. Multiway attention networks for modeling sentence pairs. In IJCAI, pages 4411– 4417. find
  20. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018. ↗
  21. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. ↗
  22. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480–1489.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fadel-2019-tha3aroon-nsurl-2019-task-8/