Elamur Research⌕
☰
home / papers / 2019 / Explicit Pairwise Word Interaction Modeling Improves Pretrained Transformers for English Semantic Similarity Tasks

Explicit Pairwise Word Interaction Modeling Improves Pretrained Transformers for English Semantic Similarity Tasks

Paper · 2019 · Yinan Zhang, Raphael Tang and Jimmy Lin

Summary

Pretrained transformers such as BERT handle word-to-word relations between two sentences only implicitly, through unconstrained self-attention. Older embedding-based similarity models instead compared every word of one sentence against every word of the other. The authors ask whether that older explicit comparison still helps a pretrained transformer. They attach the very deep pairwise word interaction module (VD-PWI, from He and Lin 2016) on top of BERT BASE. BERT's contextual vectors build a similarity cube from cosine distance, L2 distance and dot product for every cross-sentence word pair. A pairwise focus step weakens unimportant pairs, and a 19-layer CNN then classifies the resulting pattern. The whole stack is fine-tuned end-to-end. Tests cover STS-B, SICK, WikiQA and TrecQA. When both sentences are encoded jointly, the hybrid beats plain BERT on all four datasets, by 0.9 points on average, which is statistically significant. The result suggests that hand-designed interaction structure can still add value on top of large pretrained models.

Key points

  • Four English benchmarks are used. STS-B and SICK report Pearson's r and Spearman's ρ. WikiQA (12K pairs after filtering) and TrecQA (56K pairs) report MAP and MRR.
  • BERT BASE JOINT + PWIM scores TrecQA 83.7/87.9 versus 81.2/86.2 for plain BERT, and SICK 88.5/83.8 versus 87.9/82.3. STS-B stays unchanged at 84.7/83.9.
  • Separate encoding lets the sentences meet only in the PWI module. It collapses on TrecQA to about 69 MAP, below even the original non-pretrained PWIM (75.9), so the gap to joint encoding reaches roughly 14 points.
  • Removing the BiLSTM between BERT and the interaction layer makes no significant difference. Holm–Bonferroni-corrected Wilcoxon signed-rank tests support this.
  • The setup is tuned by grid search over the BERT learning rate and epochs, plus random search for the PWI part. It uses KL-divergence loss for graded scores and NLL loss for QA classification.

Interesting ideas

  • I1. Treat sentence-pair similarity as reading an image: lay out a grid of every word in one sentence against every word in the other, and let a pattern detector find strong alignment regions.
  • I2. Compare each word pair with several measures at once (cosine, Euclidean distance, dot product) instead of a single score, since each captures a different kind of closeness.
  • I3. Damp the many unimportant word pairs (here by a factor of ten) so that only the strongest alignments drive the decision.
  • I4. Letting the two inputs interact early (joint encoding) proved essential, because comparing independently built representations sometimes produced degenerate models.
  • I5. Report ablations with paired non-parametric tests (one- and two-sided Wilcoxon) and correct for multiple comparisons, rather than reading raw score gaps.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): When picking which sentence of a fable answers a question, a question-word × sentence-word alignment grid over lemmas or frame slots could give an inspectable reason for the chosen sentence.
  • I2 (world): Entity and coreference matching across mentions could combine several distinct similarity signals and keep each one visible in the explanation.
  • I3 (frames): When aligning a question frame to text frames, downweighting non-best role matches would keep the matcher focused and easier to explain.
  • I4 (answer): Question and passage should be analyzed with mutual awareness, for example question-guided frame selection, rather than as two fully separate pipelines that meet only at the end.
  • I5 (learn): Comparisons between parser or reader variants could use Wilcoxon tests with Holm–Bonferroni correction, so small gains are not mistaken for real progress.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WaterlooWaterloo · Canada44 works · 12,783 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.English · 1,373 works

Cite

Cite as Yinan Zhang, Raphael Tang and Jimmy Lin. 2019. Explicit Pairwise Word Interaction Modeling Improves Pretrained Transformers for English Semantic Similarity Tasks. arXiv:1911.02847.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1911.02847 · arXiv 1911.02847

@misc{zhang2019explicit,
  title = {{Explicit Pairwise Word Interaction Modeling Improves Pretrained Transformers for English Semantic Similarity Tasks}},
  author = {Yinan Zhang, Raphael Tang and Jimmy Lin},
  year = {2019},
  doi = {10.48550/arxiv.1911.02847},
  eprint = {1911.02847},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1911.02847}
}

References

22 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation.
  2. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling. In Fifteenth Annual Conference of the International Speech Communication Association.
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  4. Hua He, Kevin Gimpel, and Jimmy Lin. 2015. Multi-perspective sentence similarity modeling with convolutional neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
  5. Hua He and Jimmy Lin. 2016. Pairwise word interaction modeling with deep neural networks for semantic similarity measurement. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  6. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation. ↗
  7. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980. ↗
  8. Wuwei Lan and Wei Xu. 2018. Neural network models for paraphrase identification, semantic textual similarity, natural language inference, and question answering. In Proceedings of the 27th International Conference on Computational Linguistics. ↗
  9. Linqing Liu, Wei Yang, Jinfeng Rao, Raphael Tang, and Jimmy Lin. 2019a. Incorporating contextual and syntactic structures improves semantic similarity modeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. find
  10. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692. ↗
  11. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th International Workshop on Semantic Evaluation.
  12. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems. ↗
  13. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing.
  14. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  15. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. find
  16. Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-RMSProp, Coursera: Neural networks for machine learning. Technical Report. find
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems.
  18. Mengqiu Wang, Noah A. Smith, and Teruko Mitamura. 2007. What is the jeopardy model? a quasi-synchronous grammar for QA. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning.
  19. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
  20. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. arXiv:1906.08237. ↗
  21. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  22. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2019-explicit-pairwise-word-interaction-modeling/