Elamur Research⌕
☰
home / papers / 2025 / Compositionality and Sentence Meaning: Comparing Semantic Parsing and Transformers on a Challenging Sentence Similarity Dataset

Compositionality and Sentence Meaning: Comparing Semantic Parsing and Transformers on a Challenging Sentence Similarity Dataset

Paper · 2025 · James Fodor, Simon De Deyne and Shinsuke Suzuki

Summary

Compositionality is the question of how people build the meaning of a sentence from the meanings of its words. Models of it usually fall into two families. Vector-based approaches encode a sentence as a numeric embedding, while syntax-based approaches represent it as a labeled parse graph. Comparing the two has been difficult because existing STS datasets do not systematically probe compositional changes. The authors therefore built STS3k, a collection of 2,800 sentence pairs with human similarity ratings. The pairs deliberately vary how words are combined, and they include an adversarial subset. On this data the authors tested arithmetic embedding baselines, transformer sentence encoders, AMR graph-matching metrics, and hybrid systems. Among the hybrids are two new training-free models that pair semantic parses (AMR or VerbNet roles) with ConceptNet word vectors. State-of-the-art transformers track human judgments poorly on STS3k, while the simple hybrids do substantially better. Their advantage comes from mirroring human sensitivity to specific structural edits. The work suggests that combining symbolic structure with distributional vectors better reflects how humans represent compositional meaning.

Key points

  • New resource: STS3k contains 2,800 human-rated sentence pairs with adversarial and non-adversarial portions, designed to expose the strengths and weaknesses of vector, syntax, and hybrid models.
  • Models compared:
    • Arithmetic vector-based: Mean-CN, Mult-CN, Conv-CN.
    • Neural vector-based: InferSent, Universal, ERNIE 2.0, SentBERT, DefSent, OpenAI Embeddings.
    • Syntax-based: AMR-SMATCH, AMR-WWLK.
    • Hybrid: AMRBART, S3BERT, plus the newly introduced AMR-CN and VerbNet-CN.
  • VerbNet-CN results (SemParse parser): correlation with human ratings is about .67 on the full set, .65 on the non-adversarial part, and .65 on the adversarial part, achieved without task-specific training.
  • Robustness to the parser: replacing SemParse with a GPT-4 role-labeling parser gives similar scores (.673 overall), and the two similarity series correlate at 0.92.
  • Fine-tuning experiment: transformer embeddings learn the task on a random train/test split but fail to generalize when trained on non-adversarial pairs and tested on adversarial ones. VerbNet-CN stays moderately good in both settings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia100 works · 22,618 citations🎓Hitotsubashi UniversityTokyo · Japan6 works · 28 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semantic parsing & meaning representation · 185 works

Cite

Cite as James Fodor, Simon De Deyne and Shinsuke Suzuki. 2025. Compositionality and Sentence Meaning: Comparing Semantic Parsing and Transformers on a Challenging Sentence Similarity Dataset. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00536

@article{fodor2025compositionality,
  title = {{Compositionality and Sentence Meaning: Comparing Semantic Parsing and Transformers on a Challenging Sentence Similarity Dataset}},
  author = {James Fodor, Simon {De Deyne} and Shinsuke Suzuki},
  journal = {Computational Linguistics},
  year = {2025},
  doi = {10.1162/coli\_a\_00536},
  url = {https://doi.org/10.1162/coli_a_00536}
}

References

55 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdalla, Mohamed, Krishnapriya Vishnubhotla, and Saif Mohammad. 2023. What makes sentences semantically related? A textual relatedness dataset and empirical study. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 782–796. https://d… find
  2. Abend, Omri and Ari Rappoport. 2013. Universal conceptual cognitive annotation (UCCA). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228–238.
  3. Agirre, Eneko, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. 2016. SemEval-2016 Task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval-2016. 10th International Workshop on Semantic Evaluation, p…
  4. Alishahi, Afra and Suzanne Stevenson. 2010. A computational model of learning semantic roles from child-directed language. Language and Cognitive Processes, 25(1):50–93. https://doi.org/10.1080/01690960902840279 Almeida, Felipe and Geraldo Xexéo. 2019. Word embeddings: A survey. arXiv preprint arX… ↗
  5. Baker, Collin F., Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics, pages 86–90. https://doi.org/10.3115/980451 .980860 Ballesteros, Miguel, Bernd Bohnet, Simon Mille, and Leo Wanner. 2… ↗
  6. Banarescu, Laura, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Disco…
  7. Baroni, Marco. 2020. Linguistic generalization and compositionality in modern artificial neural networks. Philosophical Transactions of the Royal Society B, 375(1791):20190307. https://doi.org/10.1098/rstb.2019.0307, PubMed: 31840578 Baroni, Marco, Raffaella Bernardi, and Roberto Zamparelli. 2014.… — [doi:10.1098/rstb.2019.0307](https://doi.org/10.1098/rstb.2019.0307) ↗
  8. Batchkarov, Miroslav, Thomas Kober, Jeremy Reffin, Julie Weeds, and David Weir. 2016. A critique of word similarity as a method for evaluating distributional semantic models. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 7–12. https://doi.org /10.18653/… ↗
  9. Bevilacqua, Michele, Rexhina Blloshmi, and Roberto Navigli. 2021. One SPRING to rule them both: Symmetric AMR semantic parsing and generation without a complex pipeline. In Proceedings of AAAI, pages 12564–12573. https://doi.org/10.1609/aaai.v35i14.17489 Blacoe, William and Mirella Lapata. 2012. A… ↗
  10. Blouw, Peter, Eugene Solodkin, Paul Thagard, and Chris Eliasmith. 2016. Concepts as semantic pointers: A framework and computational model. Cognitive Science, 40(5):1128–1162. https://doi.org/10.1111/cogs.12265, PubMed: 26235459 Boleda, Gemma. 2020. Distributional semantics and linguistic theory. An… ↗
  11. Bubeck, Sébastien, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4.
  12. Cai, Shu and Kevin Knight. 2013. Smatch: An evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752. find
  13. Cai, Xingyu, Jiaji Huang, Yuchen Bian, and Kenneth Church. 2021. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference on Learning Representations. find
  14. Cer, Daniel, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Dem…
  15. Huang. 2019. A structured distributional model of sentence meaning and processing. Natural Language Engineering, 25(4):483–502. https://doi.org/10.1017/S1351324919000214 Chomsky, Noam. 1956. Three models for the description of language. IRE Transactions on Information Theory, 2(3):113–124. https://… ↗
  16. Clark, Kevin, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? An analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286. https://doi.org/10.18653/v1/W19-4828 Clark,… ↗
  17. Cordeiro, Silvio, Aline Villavicencio, Marco Idiart, and Carlos Ramisch. 2019. Unsupervised compositionality prediction of nominal compounds. Computational Linguistics, 45(1):1–57. https://doi.org/10.1162/coli a 00341 Csordás, Róbert, Kazuki Irie, and Juergen Schmidhuber. 2021. The devil is in th… ↗
  18. Dolan, Bill and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005), pages 9–16.
  19. Donatelli, Lucia and Alexander Koller. 2023. Compositionality in computational linguistics. Annual Review of Linguistics, 9. https://doi.org/10.1146/annurev -linguistics-030521-044439 Dufter, Philipp, Martin Schmitt, and Hinrich Schütze. 2022. Position information in transformers: An overview. Comp… ↗
  20. Emerson, Guy. 2020. What are the goals of distributional semantics? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7436–7453. https://doi.org/10.18653/v1/2020.acl-main.663 Erk, Katrin. 2012. Vector space models of word meaning and phrase meaning: A… ↗
  21. Fodor, Jerry and Brian P. McLaughlin. 1990. Connectionism and the problem of systematicity: Why Smolensky’s solution doesn’t work. Cognition, 35(2):183–204. https://doi.org/10.1016/0010 -0277(90)90014-B, PubMed: 2354612 Fodor, Jerry A. and Zenon W. Pylyshyn. 1988. Connectionism and cognitive archite… — [doi:10.1016/0010](https://doi.org/10.1016/0010) ↗
  22. Gajewski, Jon. 2015. Foundations of formal semantics, pages 71–89. Routledge. Geirhos, Robert, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673… — [doi:10.1038/s42256-020](https://doi.org/10.1038/s42256-020) find
  23. Golan, Tal, Matthew Siegelman, Nikolaus Kriegeskorte, and Christopher Baldassano. 2023. Testing the limits of natural language models for predicting human language judgments. Nature Machine Intelligence, 5(9):952–964. https://doi.org/10.1038/s42256-023-00718-1 Goldstone, Robert L. and Ji Yun Son. 2… ↗
  24. Grefenstette, Edward and Mehrnoosh Sadrzadeh. 2011. Experimental support for a categorical compositional distributional model of meaning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1394–1404. Greff, Klaus, Sjoerd Van Steenkiste, and Jürgen Schmidhube… find
  25. Gubelmann, Reto and Siegfried Handschuh. 2022. Uncovering more shallow heuristics: Probing the natural language inference capacities of transformer-based pre-trained language models using syllogistic patterns. arXiv preprint arXiv:2201.07614.
  26. Gung, James. 2020. SemParse VerbNet parser. https://github.com/jgung/verbnet -parser Gupta, Ashim, Giorgi Kvernadze, and Vivek Srikumar. 2021. BERT & family eat word salad: Experiments with text understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12946–12954. https:/… ↗
  27. Janssen, T. M. V. 1986. Foundations and applications of Montague grammar: 8M part 1: Philosophy, framework, computer science. find
  28. Jiang, Shuyu, Xingshu Chen, and Rui Tang. 2023. Prompt packer: Deceiving LLMs through compositional instruction with hidden attacks. arXiv preprint arXiv:2310.10077. Kabbach, Alexandre, Corentin Ribeyre, and Aurélie Herbelot. 2018. Butterfly effects in frame semantic parsing: Impact of data process… — [arXiv:2310.10077](https://arxiv.org/abs/2310.10077) find
  29. Kartsaklis, Dimitri, Mehrnoosh Sadrzadeh, and Stephen Pulman. 2013. Separating disambiguation from composition in distributional semantics. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 114–123. find
  30. Kasami, Tadao. 1966. An efficient recognition and syntax-analysis algorithm for context-free languages. Coordinated Science Laboratory Report no. R-257. find
  31. Kemp, Charles, Aaron Bernstein, and Joshua B. Tenenbaum. 2005. A generative theory of similarity. In Proceedings of the 27th Annual Conference of the Cognitive Science Society, pages 1132–1137. Keysers, Daniel, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola… find
  32. Kim, Najoung and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087–9105. https://doi.org/10.18653/v1/2020.emnlp-main.731 Kingsbury, Paul R.… ↗
  33. Lake, Brenden M., Tal Linzen, and Marco Baroni. 2019. Human few-shot learning of compositional instructions. In 41st Annual Meeting of the Cognitive Science Society: Creativity+ Cognition+ Computation, CogSci 2019, pages 611–617. find
  34. Landauer, Thomas K., Peter W. Foltz, and Darrell Laham. 1998. An introduction to latent semantic analysis. Discourse Processes, 25(2–3):259–284. https://doi.org/10.1080/01638539809545028 Lau, Jey Han, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probab… ↗
  35. Martin, Andrea E. and Giosuè Baggio. 2020. Modelling meaning composition from formalism to mechanism. Philosophical Transactions of the Royal Society B, 375(1791):1–7. https://doi.org/10.1098/rstb.2019.0298, PubMed: 31840588 Martin, Andrea E. and Leonidas A. A. Doumas. 2020. Tensors and compositio… ↗
  36. Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26:3111–3119.
  37. Mitchell, Jeff and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. https://doi.org/10.1111/j.1551-6709.2010.01106.x, PubMed: 21564253 Montague, Richard. 1970. Universal grammar. Theoria, 36:373–398. https://doi.org/10.1111/j.1755 -2567.19… ↗
  38. Opitz, Juri and Anette Frank. 2022. SBERT studies meaning representations: Decomposing sentence embeddings into explainable semantic features. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference… find
  39. O’shea, James, Zuhair Bandar, and Keeley Crockett. 2014. A new benchmark dataset with production methodology for short text semantic similarity algorithms. ACM Transactions on Speech and Language Processing (TSLP), 10(4):1–63. https://doi.org/10.1145/2537046 — [doi:10.1145/2537046](https://doi.org/10.1145/2537046) ↗
  40. Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. Padó,…
  41. Ranasinghe, Tharindu, Constantin Orǎsan, and Ruslan Mitkov. 2019. Enhancing unsupervised sentence similarity methods with deep contextualised word representations. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 994–1003. https://… find
  42. Shi, Haoyue, Hao Zhou, Jiaze Chen, and Lei Li. 2018. On tree-based neural sentence modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4631–4641. https://doi.org/10.18653/v1/D18 -1492 Simoulin, Antoine and Benoı̂t Crabbé. 2022. Unifying parsing… ↗
  43. Socher, Richard, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211…
  44. Soulos, Paul, Tom McCoy, Tal Linzen, and Paul Smolensky. 2019. Discovering the compositional structure of vector representations with role learning networks. arXiv preprint arXiv:1910.09113. https://doi.org/10.18653/v1/2020 .blackboxnlp-1.23 Srivastava, Aarohi, Abhinav Rastogi, Abhishek Rao, Abu Awa… ↗
  45. Stewart, Terrence and Chris Eliasmith. 2009. Compositionality and biologically plausible models. The Oxford Handbook of Compositionality, pages 596–615. find
  46. Su, Jianlin, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316. find
  47. Sun, Yu, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. ERNIE 2.0: A continual pre-training framework for language understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8968–8975. https://doi.org/10.1609/aaai.v34i05.6428 Szabó, Zoltá… ↗
  48. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems, 30.
  49. Vecchi, Eva M., Marco Marelli, Roberto Zamparelli, and Marco Baroni. 2017. Spicy adjectives and nominal donkeys: Capturing semantic deviance using compositionality in distributional spaces. Cognitive Science, 41(1):102–136. https://doi.org/10.1111/cogs.12330, PubMed: 26991668 Wang, Bin, C.-C. — [doi:10.1111/cogs.12330](https://doi.org/10.1111/cogs.12330) ↗
  50. Jay Kuo, and Haizhou Li. 2022. Just rank: Rethinking evaluation with word and sentence similarities. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6060–6077. find
  51. Wang, Bin, Angela Wang, Fenxiao Chen, Yuncheng Wang, and C.-C. Jay Kuo. 2019. Evaluating word embedding models: Methods and experimental results. APSIPA Transactions on Signal and Information Processing, 8:e19. https://doi.org/10 .1017/ATSIP.2019.12 Wang, Jiongxiao, Zichen Liu, Keun Hee Park, Muhao… find
  52. Wang, Tiexin, Hui Shi, Wenjing Liu, and Xinhua Yan. 2022. A joint FrameNet and element focusing sentence-BERT method of sentence similarity computation. Expert Systems with Applications, 200:117084. https://doi.org/10.1016/j .eswa.2022.117084 Wang, Zhiguo, Haitao Mi, and Abraham Ittycheriah. 2016. S… — [doi:10.1016/j](https://doi.org/10.1016/j) find
  53. Warstadt, Alex, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392. https://doi.org/10.1162/tacl a 00321 Webson, Albe… ↗
  54. Jay Kuo. 2023. SynWMD: Syntax-aware word mover’s distance for sentence similarity evaluation. Pattern Recognition Letters, 170:48–55. https://doi.org/10.1016/j.patrec.2023.04.012 Westerståhl, Dag. 1998. On mathematical proofs of the vacuity of compositionality. Linguistics and Philosophy, pages 63… ↗
  55. Zhang, Honghua, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. 2022. On the paradox of learning to reason from data. arXiv preprint arXiv:2205.11502. https://doi.org/10 .24963/ijcai.2023/375 Zhang, MeiShan. 2020. A survey of syntactic-semantic parsing based on constituent and de… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fodor-2025-compositionality-sentence-meaning-comparing-semantic/