Elamur Research⌕
☰
home / papers / 2016 / Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement

Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement

Summary

The paper tackles semantic textual similarity (STS), which means scoring how close two sentences are in meaning. The authors argue that the dominant neural "Siamese" approach compresses each sentence into a single fixed-length vector and loses word-level detail. Instead, they compare the two sentences word by word.

Their architecture has four stages:

  1. Shared Bi-LSTMs encode each word in its context.
  2. Every word context in one sentence is compared with every word context in the other, using several distance functions. The results form a 3D "similarity cube."
  3. A similarity focus layer boosts the most informative word pairs.
  4. A 19-layer deep ConvNet treats the result as a pattern recognition problem.

The model uses only pretrained word embeddings, with no sparse features, parsers, WordNet or unsupervised pretraining. Its evaluation covered two answer selection benchmarks (WikiQA, TrecQA) plus ten test sets drawn from three separate SemEval STS competitions. It reaches state-of-the-art results on all five tasks and beats the multi-perspective CNN of He et al. (2015) while being much smaller and faster. This shows that explicit fine-grained word alignment can replace both heavy feature engineering and wide, costly architectures.

Key points

  • Similarity cube. Each word pair gets 13 values: 12 similarity scores plus a padding indicator. The 12 scores come from cosine, L2 Euclidean and dot product, each applied to four views of the hidden states: the full Bi-LSTM state, the forward state, the backward state, and the sum of forward and backward.

  • Reported results:

    DatasetMetricScore
    SICKPearson0.8784
    WikiQAMAP / MRR0.7090 / 0.7234
    TrecQAMAP / MRR0.7588 / 0.8219

    On STS2014, the model compares favorably with the winning feature-engineered system (Sultan et al., 2014).

  • Ablation results:

    • Removing context modeling causes the biggest drop (−0.1225 Pearson on SICK, −0.0990 MRR on WikiQA).
    • Removing the focus layer costs −0.0083 on SICK and −0.0327 on WikiQA.
    • Replacing the focus layer with dropout (p = 0.3) hurts even more than removing it (−0.0314 / −0.0403).

  • Efficiency on SICK versus He et al. (2015):

    • Parameters: 1.7M vs 10.0M, about 83% fewer.
    • Average epoch time on one CPU thread: 664 s vs 2265 s, about 3.4× faster.

  • Interpretability: visualizing the cosine channel of the focusCube on SICK pairs shows the model assigns sharply higher values to semantically matching word pairs than to their neighbors.

Interesting ideas

  • I1. Treat sentence comparison as image recognition. Stacking pairwise similarities into a multi-channel grid lets a standard deep ConvNet find alignment patterns. The same trick could suit any pairwise text task, such as paraphrase detection, cross-lingual matching or entity matching.
  • I2. Use several similarity metrics side by side. Cosine captures angle, while L2 and dot product capture magnitude. Feeding all three, over forward, backward, combined and summed states, gives the downstream network richer evidence than any single score.
  • I3. Compare contexts, not raw words. Running the comparison on contextual encoder states rather than embeddings proved to be the most important component in the ablations. This argues for contextualizing before matching in alignment-style models.
  • I4. Use a dropout control in ablations. Swapping the focus layer for random dropout tests whether targeted reweighting matters, rather than just any perturbation. It is a useful control design for evaluating attention-like mechanisms.
  • I5. Aim for lean models without external resources. Strong results without parsers or WordNet make the approach attractive for lower-resource languages, where such tools may be missing or weak. This is especially relevant for Ukrainian NLP work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Institutions

🎓University of Maryland, College ParkCollege Park · United States65 works · 21,800 citations🎓University of WaterlooWaterloo · Canada36 works · 11,969 citations

Cite

Cite as Hua He and Jimmy Lin. 2016. Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1108

@inproceedings{he2016pairwise,
  title = {{Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement}},
  author = {Hua He and Jimmy Lin},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1108},
  url = {https://doi.org/10.18653/v1/n16-1108}
}

References

55 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics, pages 385–393. our card
  2. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation, pages 81–… our card
  3. Yaser Al-onaizan, Jan Curin, Michael Jahr, Kevin Knight, John Lafferty, Dan Melamed, Franz-Josef Och, David Purdy, Noah A. Smith, and David Yarowsky. 1999. Statistical machine translation. Final report, JHU Summer Workshop on Language Engineering. find
  4. Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. 2013. Deep canonical correlation analysis. In Proceedings of the 30th International Conference on Machine Learning, pages 1247–1255. find
  5. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. Daniel Bär, Chris Biemann, Iryna Gurevych, and Torsten Zesch. 2012. UKP: Computing semantic textual similarity by combining multiple content similari… our card
  6. Islam Beltagy, Katrin Erk, and Raymond Mooney. 2014. Probabilistic soft logic for semantic textual similarity. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics, pages 1210– 1219. find
  7. Johannes Bjerva, Johan Bos, Rob van der Goot, and Malvina Nissim. 2014. The meaning factory: Formal semantics for recognizing textual entailment and determining semantic similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation, pages 642–646. Léon Bottou. 1998. Online lea… find
  8. Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd International Conference on Machine Learning, pages 89–96. our card
  9. Hang Cui, Renxu Sun, Keya Li, Min-Yen Kan, and Tat-Seng Chua. 2005. Question answering passage retrieval using dependency relations. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 400–407. find
  10. Dipanjan Das and Noah A. Smith. 2009. Paraphrase identification as probabilistic quasi-synchronous recognition. In Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics, pages 468–476. our card
  11. Christiane Fellbaum. 1998. find
  12. WordNet: An Electronic Lexical Database. MIT Press. Samuel Fern and Mark Stevenson. 2008. A semantic similarity approach to paraphrase detection. In Proceedings of the 11th Annual Research Colloquium of the UK Special-Interest Group for Computational Lingusitics, pages 45–52. find
  13. Katja Filippova, Enrique Alfonseca, Carlos A. Colmenares, Lukasz Kaiser, and Oriol Vinyals. 2015. Sentence compression by deletion with LSTMs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 360– 368. our card
  14. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The Paraphrase Database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 758–764. Alex Graves, Santiago Fernández, an… our card
  15. Hua He, Kevin Gimpel, and Jimmy Lin. 2015. Multi-perspective sentence similarity modeling with convolutional neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1576–1586. our card
  16. Michael Heilman and Noah A. Smith. 2010. Tree edit models for recognizing textual entailments, paraphrases, and answers to questions. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 1011–1019. Sepp Hochr… our card
  17. Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In Advances in Neural Information Processing Systems 27, pages 2042–2050. find
  18. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM International Conference on Information & Knowledge Management, pages 2333–2338. Sergio Jimenez, George… find
  19. Abhay Kashyap, Lushan Han, Roberto Yus, Jennifer Sleeman, Taneeya Satyapanich, Sunil Gandhi, and Tim Finin. 2014. Meerkat mafia: Multilingual and cross-level semantic textual similarity systems. In Proceedings of the 8th International Workshop on Semantic Evaluation, pages 416–423. find
  20. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, pages 1097–1105. find
  21. Alice Lai and Julia Hockenmaier. 2014. Illinois-LH: A denotational and distributional approach to semantics. In Proceedings of the 8th International Workshop on Semantic Evaluation, pages 329–334. find
  22. Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31th International Conference on Machine Learning, pages 1188–1196. our card
  23. Jimmy Lin. 2007. An exploration of the principles underlying redundancy-based factoid question answering. ACM Transactions on Information Systems, 25(2):1–55. André Lynum, Partha Pakray, Björn Gambäck, and Sergio Jimenez. 2014. NTNU: Measuring semantic similarity with sublexical feature represent… find
  24. Nitin Madnani, Joel Tetreault, and Martin Chodorow. 2012. Re-examining machine translation metrics for paraphrase identification. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 182–190. our card
  25. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th Internatio… our card
  26. Raymond J. Mooney. 2014. Semantic parsing: Past, present, and future. In ACL Workshop on Semantic Parsing. Presentation slides. find
  27. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543. our card
  28. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. CoRR, abs/1509.00685. Frane Šarić, Goran Glavaš, Mladen Karan, Jan Šnajder, and Bojana Dalbelo Bašić. 2012. TakeLab: systems for measuring semantic text similarity. In Proc… find
  29. Aliaksei Severyn and Alessandro Moschitti. 2013. Automatic feature engineering for answer selection and extraction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 458–467. find
  30. Aliaksei Severyn and Alessandro Moschitti. 2015. Learning to rank short text pairs with convolutional deep neural networks. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 373– 382. find
  31. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556. our card
  32. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems 24, pages 801–809. find
  33. Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2:207– 218. our card
  34. Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014. Dls@cu: Sentence similarity from word alignment. In Proceedings of the 8th International Workshop on Semantic Evaluation, pages 241–246. find
  35. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27, pages 3104–3112. find
  36. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–9. find
  37. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 1556– 1566. find
  38. Tijmen Tieleman and Geoffrey E. Hinton. 2012. Lecture 6.5—RMSProp: Divide the gradient by a running average of its recent magnitude. Coursera: Neural Networks for Machine Learning. find
  39. Stephen Wan, Mark Dras, Robert Dale, and Cecile Paris. 2006. Using Dependency-based Features to Take the “Para-farce” out of Paraphrase. In Australasian Language Technology Workshop, pages 131–138. find
  40. Mengqiu Wang and Christopher D. Manning. 2010. Probabilistic tree-edit models with structured latent variables for textual entailment and question answering. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1164–1172. find
  41. Di Wang and Eric Nyberg. 2015. A long short-term memory model for answer sentence selection in question answering. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 707–712. find
  42. Mengqiu Wang, Noah A. Smith, and Teruko Mitamura. 2007. What is the Jeopardy model? A quasi-synchronous grammar for QA. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 22–32. our card
  43. Jason Weston, Samy Bengio, and Nicolas Usunier. 2011. Wsabie: Scaling up to large vocabulary image annotation. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence, pages 2764– 2770. find
  44. John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Dan Roth. 2015. From paraphrase database to compositional paraphrase model and back. Transactions of the Association for Computational Linguistics, 3:345–358. our card
  45. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. In Proceedings of the 4th International Conference on Learning Representations. our card
  46. Ronald J. Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270–280. find
  47. Wei Xu, Alan Ritter, Chris Callison-Burch, William B. Dolan, and Yangfeng Ji. 2014. Extracting lexically divergent paraphrases from Twitter. Transactions of the Association for Computational Linguistics, 2:435– 448. find
  48. Wei Xu. 2014. Data-Drive Approaches for Paraphrasing Across Language Variations. Ph.D. thesis, Department of Computer Science, New York University. find
  49. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018. our card
  50. Xuchen Yao, Benjamin Van Durme, Chris Callison-burch, and Peter Clark. 2013. Answer extraction as sequence tagging with tree edit distance. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 858–867… find
  51. Wentau Yih, Ming-Wei Chang, Christopher Meek, and Andrzej Pastusiak. 2013. Question answering using enhanced lexical semantic models. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1744–1753. Wenpeng Yin and Hinrich Schütze. 2015. Convolutional neu… find
  52. Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. 2014. Deep Learning for Answer Sentence Selection. In NIPS Deep Learning Workshop. find
  53. Guido Zarrella, John Henderson, Elizabeth M. Merkhofer, and Laura Strickhart. 2015. MITRE: Seven systems for semantic similarity in tweets. In Proceedings of the 9th International Workshop on Semantic Evaluation, pages 12–17. find
  54. Jiang Zhao, Tian Tian Zhu, and Man Lan. 2014. ECNU: One stone two birds: Ensemble of heterogenous measures for semantic relatedness and textual entailment. In Proceedings of the 8th International Workshop on Semantic Evaluation, pages 271–277. find
  55. Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. 2015. Long short-term memory over recursive structures. In Proceedings of the 32nd International Conference on Machine Learning, pages 1604–1612. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/he-2016-pairwise-word-interaction-modeling-deep/