Elamur Research⌕
☰
home / papers / 2024 / Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics

Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics

Paper · 2024 · Fangru Lin, Daniel Altshuler and Janet B. Pierrehumbert

Summary

The paper asks whether large language models know two different things about scalar adjectives such as likely/certain or warm/scalding. The first is lexical semantics: which scale an adjective belongs to and how intense it is on that scale. The second is scalar diversity: some weaker adjectives readily suggest "not the stronger one" (likely → not certain), while others rarely do (good does not suggest not excellent). The authors test eight models from three families: encoder (BERT, RoBERTa), decoder (Falcon-7B-instruct, GPT-4) and encoder-decoder (Flan-T5-xl/xxl). They use three scalar adjective datasets (DEMELO, CROWD, WILKINSON) and context sentences from ukWac. Open models are probed directly through contextual embeddings. All models are also probed indirectly through text templates and prompts, and the direct results on open models serve as a check on the indirect method. The models turn out to hold substantial lexical knowledge about scales but reproduce human scalar diversity poorly. Bigger models are not reliably better. This matters because knowing word meanings does not guarantee pragmatic inference.

Key points

  • The datasets contain 185 half-scales in total:
    • DM: 87 half-scales, 548 adjective pairs
    • CD: 77 half-scales, 330 pairs
    • WK: 21 half-scales, 61 pairs
    • ukWac supplies 10 shared context sentences per half-scale, and substitution acceptability was checked with context2vec.
  • In direct probing of scale membership, a scale vector is built by adding the embeddings of the weakest and strongest adjectives. Scales are then ranked by cosine similarity to each adjective's embedding.
  • Indirect probing uses Google Ngram as a baseline and 34 intensity templates: 22 weak→strong (e.g. "X but not Y", "not just X but Y") and 12 strong→weak (e.g. "not Y, just X").
  • GPT-4 scored best on indirect intensity ranking: DM 0.666, CD 0.739, WK 0.852.
  • Scalar diversity was evaluated on three human-judgment datasets (RX, GZ, PVT). Flan-T5-xxl averaged about 0.81, well above Falcon. Scale size did not predict performance.

Interesting ideas

  • I1. Coordination frames such as "X or even Y" and "X, if not Y" read well only for same-scale pairs ordered weak→strong, so a single construction tests both scale membership and ordering.
  • I2. A scale is summarized by combining only its two endpoint words, which makes a cheap anchor for assigning new adjectives to a scale.
  • I3. Indirect (prompt-based) probes are trusted only after their model rankings agree with direct (representation-based) probes on open models.
  • I4. To avoid picking prompts that overfit the test data, the best template is chosen on the other datasets and then applied to the held-out one.
  • I5. In forced-choice prompts, the order of the correct answer is shuffled and a "none" option is offered for equal-intensity pairs, which removes position bias and lets ties be expressed.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Such frames could be mined over a corpus to fill and order the compiled measures/values scales, so that each ordering carries an inspectable textual justification.
  • I2 (world): Endpoint anchors could place adjectives that WordNet leaves unscaled onto existing ontology scales, flagged as low-confidence until confirmed.
  • I3 (learn): LLM teacher tiers could be validated the same way, accepting a teacher's prompt-based labels only where they track an independent, inspectable signal on a calibration subset.
  • I4 (learn): Evaluation could apply the same held-out selection to repair rules or prompt templates, so that reported scores are not inflated by tuning on the test set.
  • I5 (learn): Silver-data queries to LLM teachers could randomize option order and always allow an abstain/equal answer, which also gives a natural negative control.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksInterpretability · 130 worksLexical semantics & word embeddings · 63 works

Cite

Cite as Fangru Lin, Daniel Altshuler and Janet B. Pierrehumbert. 2024. Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2404.03301

@inproceedings{lin2024probing,
  title = {{Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics}},
  author = {Fangru Lin, Daniel Altshuler and Janet B. Pierrehumbert},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/28thzn779nt3},
  eprint = {2404.03301},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.63317/28thzn779nt3}
}

References

64 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. The falcon series of open language models. arXiv preprint arXiv:2311.16867. ↗
  2. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling Instruction-Finetuned Language Models. arXiv preprint arXiv:2210.11416. ↗
  3. Anne Cocos, Skyler Wharton, Ellie Pavlick, Marianna Apidianaki, and Chris Callison-Burch. 2018. Learning Scalar Adjective Intensity from Paraphrases. In Proceedings of the 2018 Conference Some datasets containing SI triggers such as determiners and verbs, but they are also small (Ronai and Xiang, 2023b; Van Tiel et al., 2014). on Empirical Methods in Natural Language Processing, pages 1752–1762, Brussels, Belgium. Association for Computational Linguistics.
  4. Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2010. “Was It Good? It Was Provocative.” Learning the Meaning of Scalar Adjectives. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 167–176, Uppsala, Sweden. Association for Computational Linguistics.
  5. Gerard de Melo and Mohit Bansal. 2013. Good, Great, Excellent: Global Inference of Semantic Intensities. Transactions of the Association for Computational Linguistics, 1:279–290. ↗
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  7. Felix Drinkall, Eghbal Rahimikia, Janet B Pierrehumbert, and Stefan Zohren. 2024. Time Machine GPT. In NAACL 2024 Findings. find
  8. Allyson Ettinger. 2020. What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models. Transactions of the Association for Computational Linguistics, 8:34–48. ↗
  9. Wangshu Feng, Hongbo Yu, and Xiaolin Zhou. 2021. Understanding particularized and generalized conversational implicatures: Is theory-of-mind necessary? Brain and Language, 212:104878. ↗
  10. Aina Garí Soler and Marianna Apidianaki. 2020. BERT Knows Punta Cana is not just beautiful, it’s gorgeous: Ranking Scalar Adjectives with Contextualised Representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7371–7385, Online. Association for Computational Linguistics. ↗
  11. Nicole Gotzner and Diana Mazzarella. 2021. Face Management and Negative Strengthening: The Role of Power Relations, Social Distance, and Gender. Frontiers in psychology, 12:602977. ↗
  12. Nicole Gotzner, Stephanie Solt, and Anton Benz. 2018. Scalar Diversity, Negative Strengthening, and Adjectival Semantics. Frontiers in Psychology, 9. ↗
  13. Herbert P Grice. 1975. Logic and Conversation. In Speech acts, pages 41–58. Brill. ↗
  14. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  15. Jennifer Hu, Roger Levy, Judith Degen, and Sebastian Schuster. 2023. Expectations over Unspoken Alternatives Predict Pragmatic Inferences. Transactions of the Association for Computational Linguistics, 11:885–901. ↗
  16. Jennifer Hu, Roger Levy, and Sebastian Schuster. 2022. Predicting scalar diversity with context-driven uncertainty over alternatives. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 68–74, Dublin, Ireland. Association for Computational Linguistics.
  17. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are Natural Language Inference Models IMPPRESsive? Learning IM-Plicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. find
  18. Hans Kamp. 1975. Two theories about adjectives. In Formal Semantics of Natural Language. Cambridge University Press. ↗
  19. Joo-Kyung Kim and Marie-Catherine de Marneffe. 2013. Deriving adjectival scales from continuous space word representations. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1625– 1630. find
  20. Emanuele La Malfa, Christoph Weinhuber, Orazio Torre, Fangru Lin, Anthony Cohn, Nigel Shadbolt, and Michael Wooldridge. 2024. Code Simulation Challenges for Large Language Models. arXiv preprint arXiv:2401.09074. ↗
  21. Fangru Lin, Emanuele La Malfa, Valentin Hofmann, Elle Michelle Yang, Anthony Cohn, and Janet B Pierrehumbert. 2024. Graph-enhanced Large Language Models in Asynchronous Plan Reasoning. arXiv preprint arXiv:2402.02805. ↗
  22. Yuri Lin, Jean-Baptiste Michel, Erez Aiden Lieberman, Jon Orwant, Will Brockman, and Slav Petrov. 2012. Syntactic annotations for the Google Books NGram corpus. In Proceedings of the ACL 2012 System Demonstrations, pages 169–174, Jeju Island, Korea. Association for Computational Linguistics. find
  23. Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2023a. We’re Afraid Language Models Aren’t Modeling Ambiguity. arXiv preprint arXiv:2304.14399. ↗
  24. Wei Liu, Ming Xiang, and Nai Ding. 2023b. Adjective Scale Probe: Can Language Models Encode Formal Semantics Information? Proceedings of the AAAI Conference on Artificial Intelligence, 37(11):13282–13290. find
  25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. ↗
  26. Isabelle Lorge and Janet Pierrehumbert. 2023. Not Wacky vs. Definitely Wacky: A Study of Scalar Adverbs in Pretrained Language Models. pages 296–316. find
  27. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning Generic Context Embedding with Bidirectional LSTM. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 51–61, Berlin, Germany. Association for Computational Linguistics.
  28. William Merrill, Zhaofeng Wu, Norihito Naka, Yoon Kim, and Tal Linzen. 2024. Can You Learn Semantics Through Next-Word Prediction? The Case of Entailment. arXiv preprint arXiv:2402.13956. ↗
  29. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in Pre-Training Distributed Word Representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). ↗
  30. Elizabeth Pankratz and Bob van Tiel. 2021. The role of relevance for scalar diversity: a usage-based approach. Language and Cognition, 13(4):562–594. ↗
  31. Carita Paradis. 1998. Degree Modifiers of Adjectives in Spoken British English. find
  32. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language Models as Knowledge Bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463–2473, Hong Kong, China. Association for Computational Linguistics. ↗
  33. Christopher Potts. 2015. Presupposition and Implicature. The handbook of contemporary semantic theory, pages 168–202. ↗
  34. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language Models Are Unsupervised Multitask Learners. OpenAI blog, 1(8):9. find
  35. Eszter Ronai and Ming Xiang. 2022. Three Factors in Explaining Scalar Diversity. In Proceedings of Sinn und Bedeutung, volume 26, pages 716– 733. ↗
  36. Eszter Ronai and Ming Xiang. 2023a. Degree estimates as a measure of inference calculation. Proceedings of the Linguistic Society of America, 8:5537. ↗
  37. Eszter Ronai and Ming Xiang. 2023b. Memory Versus Expectation: Processing Relative Clauses in a Flexible Word Order Language. Cognitive Science, 47(1):e13227. ↗
  38. Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2024. The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs. Advances in Neural Information Processing Systems, 36. ↗
  39. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked Language Model Scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics.
  40. Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022. Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3762–3780, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  41. Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020. Harnessing the Linguistic Signal to Predict Scalar Inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5387–5403, Online. Association for Computational Linguistics. ↗
  42. Chaitanya Shivade, Marie-Catherine de Marneffe, Eric Fosler-Lussier, and Albert M Lai. 2015. Corpus-Based Discovery of Semantic Intensity Scales. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–493. find
  43. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality over a Sentiment Treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.
  44. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971. ↗
  45. Bob Van Tiel, Emiel Van Miltenburg, Natalia Zevakhina, and Bart Geurts. 2014. Scalar Diversity. Journal of Semantics, 33(1):137–175. find
  46. Jesse Vig. 2019. A Multiscale Visualization of Attention in the Transformer Model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 37–42, Florence, Italy. Association for Computational Linguistics.
  47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems, 35:24824–24837.
  48. Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, and Hinrich Schütze. 2022. The better your Syntax, the better your Semantics? Probing Pretrained Language Models for the English Comparative Correlative. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10859–10882, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  49. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv preprint arXiv:2302.11382. ↗
  50. Bryan Wilkinson and Oates Tim. 2016. A Gold Standard for Scalar Adjectives. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2669– 2675, Portorož, Slovenia. European Language Resources Association (ELRA).
  51. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  52. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗
  53. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate Before Use: Improving Few-shot Performance of Language Models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR. ↗
  54. Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual Probing Is [MASK]: Learning vs. Learning to Recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017–5033, Online. Association for Computational Linguistics. 10.
  55. Language Resource References Cocos, Anne and Wharton, Skyler and Pavlick, Ellie and Apidianaki, Marianna and Callison-Burch, Chris. 2018. Learning Scalar Adjective Intensity from Paraphrases. Association for Computational Linguistics. PID https://github.com/acocos/scalar-adj/. ↗
  56. de Melo, Gerard and Bansal, Mohit. 2013. Good, Great, Excellent: Global Inference of Semantic Intensities. MIT Press. PID http://demelo.org/gdm/intensity/. ↗
  57. Ganitkevitch, Juri and Van Durme, Benjamin and Callison-Burch, Chris. 2013. PPDB: The Paraphrase Database. Association for Computational Linguistics. PID http://paraphrase.org//. ↗
  58. Garí Soler, Aina and Apidianaki, Marianna. 2020. BERT Knows Punta Cana is not just beautiful, it’s gorgeous: Ranking Scalar Adjectives with Contextualised Representations. Association for Computational Linguistics. PID https://github.com/ainagari/scalar_adjs/tree/master/data. ↗
  59. Gotzner, Nicole and Solt, Stephanie and Benz, Anton. 2018. Scalar Diversity, Negative Strengthening, and Adjectival Semantics. PID https://github.com/jennhu/expectations-over-alternatives/blob/master/cross-scale/human_data/g18.csv. ↗
  60. Gross, Derek and Miller, Katherine J. 1990. Adjectives in WordNet. Oxford University Press, ISLRN 379-473-059-273-1. ↗
  61. Pankratz, Elizabeth and van Tiel, Bob. 2021. The role of relevance for scalar diversity: a usage-based approach. Cambridge University Press. PID https://github.com/jennhu/expectations-over-alternatives/blob/master/cross-scale/human_data/pvt21.csv. ↗
  62. Pavlick, Ellie and Rastogi, Pushpendre and Ganitkevitch, Juri and Van Durme, Benjamin and Callison-Burch, Chris. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. Association for Computational Linguistics. PID http://paraphrase.org//. ↗
  63. Ronai, Eszter and Xiang, Ming. 2022. Three factors in explaining scalar diversity. PID https://github.com/jennhu/expectations-over-alternatives/blob/master/cross-scale/human_data/rx22.csv. ↗
  64. Wilkinson, Bryan and Tim, Oates. 2016. A Gold Standard for Scalar Adjectives. European Language Resources Association (ELRA), ISLRN 691-938-401-573-9. 11.1. Alternative Methods for Scale Membership Probing

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lin-2024-probing-large-language-models-scalar/