Elamur Research⌕
☰
home / papers / 2019 / Deep Contextualized Pairwise Semantic Similarity for Arabic Language Questions

Deep Contextualized Pairwise Semantic Similarity for Arabic Language Questions

Summary

The paper addresses question-to-question (Q2Q) semantic similarity for Arabic. Two questions count as equivalent when they would receive the same answer. The task is hard because Arabic has little labelled Q2Q data, a rich morphology, and many dialects that differ from Modern Standard Arabic (MSA).

The authors release a large MSA question-pair dataset annotated at Mawdoo3. They also derive a dialect test set from the MADAR parallel corpora. They train their own ELMo model, which has a character-level CNN input and uses a corpus mixing MSA and dialect text. On top of these embeddings they compare two pair classifiers: DPAD, which works on sentence vectors, and a focus layer, which compares every word with every other word.

Training uses only MSA pairs. The best MSA model reaches 0.93 F1. The ELMo + focus layer combination gives the best dialect score, 0.82 F1, while staying strong on MSA. The work shows that a model can carry over to unseen dialects without any labelled dialect data.

Key points

  • Datasets:
    • Mawdoo3 Q2Q is a binary-labelled set of written factoid question pairs, described as the largest Arabic Q2Q dataset.
    • The dialect evaluation set is built from MADAR, which covers 24 major Arabic dialects.
  • Embeddings: the custom ELMo uses a char-CNN input and a 2-layer biLSTM. It outperforms the AraVec word embeddings.
  • F1 scores (MSA / dialect):
    • Baseline (weighted Word2vec + DPAD): 0.72 / 0.42.
    • ELMo + TrainableLSTM + DPAD: 0.93 / 0.69.
    • ELMo + TrainableLSTM + FocusLayer: 0.90 / 0.82.
  • Generalization gap: the average MSA-to-dialect F1 drop is 0.073 for focus-layer models and 0.233 for DPAD models.
  • Why dialect is harder: dialect questions are shorter, follow spoken word order, and are conversational rather than factoid.

Interesting ideas

  • I1. The authors train only on the standard language and test on varieties the model never saw, using the gap between the two scores as a robustness measure.
  • I2. The focus layer compares every word with every other word instead of squeezing each question into one vector, so differences in length and word order hurt much less.
  • I3. Using a frozen, randomly initialized LSTM under a trainable comparison layer acted as a regularizer and nearly closed the MSA-to-dialect gap.
  • I4. Character-level input lets the embeddings handle prefixes, suffixes, and unseen word forms without a separate vector for every inflection.
  • I5. Two questions are defined as equivalent when they share the same answer, rather than when their wording is similar.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): we could build test sets of informal or regional English rewrites of the same questions and report the score drop as a standing evaluation metric.
  • I2 (answer): a reader could match a question against facts in the world by aligning tree nodes or frame slots pair by pair, which also leaves an explainable alignment table.
  • I3 (tree): when training gparse on one treebank, we could test freezing part of the encoder to see whether it improves accuracy on other domains or UD dialects.
  • I4 (tokens): character features could back off for unknown words in the tagger and lemmatizer, which matters for typos, rare derivations, and new coinages.
  • I5 (answer): paraphrase detection could check whether two questions compile into the same query over the world of the text, which yields an exact and verifiable criterion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Semitic languages

Cite

Cite as Hesham Al-Bataineh, Wael Farhan, Ahmad Mustafa, Haitham Seelawi and Hussein T. Al-Natsheh. 2019. Deep Contextualized Pairwise Semantic Similarity for Arabic Language Questions. arXiv:1909.09490.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1909.09490 · arXiv 1909.09490

@misc{albataineh2019deep,
  title = {{Deep Contextualized Pairwise Semantic Similarity for Arabic Language Questions}},
  author = {Hesham Al-Bataineh, Wael Farhan, Ahmad Mustafa, Haitham Seelawi and Hussein T. Al-Natsheh},
  year = {2019},
  doi = {10.48550/arxiv.1909.09490},
  eprint = {1909.09490},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.09490}
}

References

38 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Learning semantic textual similarity from conversations. arXiv preprint arXiv:1804.07754, 2018. ↗
  2. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv preprint arXiv:1803.11175, 2018. ↗ our card
  3. Marwah Alian and Arafat Awajan. Arabic semantic similarity approaches-review. In 2018 International Arab Conference on Information Technology (ACIT), pages 1–6. IEEE, 2018. find
  4. Mahmoud Al-Ayyoub, Aya Nuseir, Kholoud Alsmearat, Yaser Jararweh, and Brij Gupta. Deep learning for arabic nlp: A survey. Journal of computational science, 26:522–531, 2018. find
  5. Haitham Seelawi, Ahmad Mustafa, Al-Bataineh Hesham, Wael Farhan, and Hussein T. Al-Natsheh. NSURL-2019 task 8: Semantic question similarity in arabic. In Proceedings of the first International Workshop on NLP Solutions for Under Resourced Languages, NSURL ’19, Trento, Italy, September 2019. find
  6. Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, et al. Unified guidelines and resources for arabic dialect orthography. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), 2018. our card
  7. Abu Bakr Soliman, Kareem Eissa, and Samhaa R. El-Beltagy. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256 – 265, 2017. Arabic Computational Linguistics. find
  8. El Moatez Billah Nagoudi and Didier Schwab. Semantic similarity of arabic sentences with word embeddings. In Proceedings of the Third Arabic Natural Language Processing Workshop, WANLP 2017@EACL, Valencia, Spain, April 3, 2017, pages 18–24, 2017. find
  9. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. ↗ our card
  10. Mohamed A Zahran, Ahmed Magooda, Ashraf Y Mahgoub, Hazem Raafat, Mohsen Rashwan, and Amir Atyia. Word representations in vector space and their applications for arabic. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 430–443. Springer, 2015. find
  11. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055, 2017. ↗ our card
  12. Yang Shao. HCTI at SemEval-2017 task 1: Use convolutional neural network to evaluate semantic textual similarity. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 130–133, 2017. find
  13. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 2333–2338. ACM, 2013. find
  14. Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014. our card
  15. Yann LeCun, Patrick Haffner, Léon Bottou, and Yoshua Bengio. Object recognition with gradient-based learning. In Shape, contour and grouping in computer vision, pages 319–345. Springer, 1999. find
  16. Dominik Scherer, Andreas Müller, and Sven Behnke. Evaluation of pooling operations in convolutional architectures for object recognition. In International conference on artificial neural networks, pages 92–101. Springer, 2010. find
  17. Rana Aref Salama, Abdou Youssef, and Aly Fahmy. Morphological word embedding for arabic. Procedia computer science, 142:83–93, 2018. find
  18. Xiao Li and Qingsheng Li. Calculation of sentence semantic similarity based on syntactic structure. Mathematical Problems in Engineering, Vol. 2015, Article ID 203475, 2015, 2015. find
  19. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.05250, 2016. ↗ our card
  20. Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge. CoRR, abs/1808.07042, 2018. ↗ our card
  21. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. The madar arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), 2018. find
  22. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. our card
  23. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States., pages 3111–3119, 2013. our card
  24. Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. Efficient non-parametric estimation of multiple embeddings per word in vector space. arXiv preprint arXiv:1504.06654, 2015. ↗ our card
  25. Heeyoul Choi, Kyunghyun Cho, and Yoshua Bengio. Context-dependent word representation for neural machine translation. Computer Speech & Language, 45:149–160, 2017. find
  26. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. ↗ our card
  27. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018. ↗ our card
  28. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  29. John Wieting and Douwe Kiela. No training required: Exploring random encoders for sentence classification. arXiv preprint arXiv:1901.10444, 2019. ↗
  30. Thomas M Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE transactions on electronic computers, (3):326–334, 1965. find
  31. Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features. In NAACL 2018 - Conference of the North American Chapter of the Association for Computational Linguistics, 2018. our card
  32. John Wieting and Kevin Gimpel. ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462, Melbourne, Australia, July 2018. Association for Computational Linguistics. our card
  33. Basant Agarwal, Heri Ramampiaro, Helge Langseth, and Massimiliano Ruocco. A deep network model for paraphrase detection in short text messages. Information Processing & Management, 54(6):922 – 937, 2018. find
  34. Hua He and Jimmy Lin. Pairwise word interaction modeling with deep neural networks for semantic similarity measurement. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 937–948, San Diego, California, June 2016. Association for Computational Linguistics. our card
  35. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16), pages 265–283, 2016. find
  36. Adam Paszke, Sam Gross, Soumith Chintala, and Gregory Chanan. Pytorch: Tensors and dynamic neural networks in python with strong gpu acceleration. PyTorch: Tensors and dynamic neural networks in Python with strong GPU acceleration, 6, 2017. find
  37. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough-to-beat baseline for sentence embeddings. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. find
  38. Noam Chomsky. Deep structure, surface structure, and semantic interpretation. Indiana University Linguistics Club, 1969. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/al-bataineh-2019-deep-contextualized-pairwise-semantic-similarity/