Elamur Research⌕
☰
home / papers / 2022 / Pre-trained Language Models' Interpretation of Evaluativity Implicature: Evidence from Gradable Adjectives Usage in Context

Pre-trained Language Models' Interpretation of Evaluativity Implicature: Evidence from Gradable Adjectives Usage in Context

Paper · 2022 · Yan Cong

Summary

This paper asks whether pre-trained language models grasp evaluativity (EVAL), a subtle pragmatic inference. If a speaker says "Maria is tall", listeners take it that Maria counts as tall by the speaker's standard. A comparative or equative ("taller than", "as tall as") carries no such commitment, since both people could be short. The author reuses the human-validated stimuli of Brasoveanu and Rett (2018): 32 gradable antonym pairs, half relative and half absolute, each placed in positive (POS), equative (EQ) and comparative (COMP) frames. GPT-3 and Macaw complete question-style prompts under three framings: a Detective reporting to a Police Chief, a "nerdy" speaker, and a "chill" speaker. The persona framings follow Beltrama and Schwarz (2021). The score is the embedding cosine similarity between each model answer and the POS counterpart of the stimulus. The models match humans in two ways: polarity has no effect, and POS is the most evaluative construction. Unlike humans, the models treat absolute adjectives as more evaluative than relative ones in nearly every condition. The study shows how datasets from experimental pragmatics can serve as probes for models.

Key points

  • The stimuli are 32 antonym pairs (16 relative, 16 absolute) in 3 constructions, giving 96 test strings. Human baselines come from Brasoveanu and Rett (2018, N=95), and the persona descriptions come from Beltrama and Schwarz (2021, N=240).
  • Two models were tested: Macaw-large (770M parameters, 24 layers, built on T5) and GPT-3/InstructGPT (175B parameters, 96 layers). Both ran as plain completion without fine-tuning, with max_tokens=100 and presence/frequency penalties set to 0.6.
  • The measure was cosine similarity between text-similarity-babbage-001 embeddings of the model's reply and the POS version of the target. It operationalizes the Bierwisch Test, under which a construction is evaluative iff it contextually entails its POS counterpart.
  • GPT-3 tracked human construction sensitivity better and behaved similarly across prompt framings. Macaw reacted more strongly to framing: the chill persona clearly strengthened its treatment of POS as evaluative.
  • On adjective class, the models mostly deviated from humans by rating relative adjectives as less evaluative than absolute ones. This held even in EQ and COMP, and the difference was significant for some Macaw settings.

Interesting ideas

  • I1. The Bierwisch Test gives a clean diagnostic: a construction carries evaluative force exactly when, in context, it entails the bare positive form. This turns a fuzzy pragmatic notion into a checkable entailment relation.
  • I2. Gradable adjectives should be split into relative ones (with a context-dependent standard, like tall or heavy) and absolute ones (anchored to a scale endpoint, like full or empty), because the class changes how evaluative inferences behave.
  • I3. The speaker's persona is treated as an experimental variable, on the idea that a literal-minded speaker invites literal readings while a socially savvy one invites implicatures. This builds speaker identity directly into interpretation.
  • I4. The author admits that a similarity score can rate an entailed but differently worded answer as dissimilar, and a non-entailed but similar answer as close. The proposed fix is to switch to entailment checking.
  • I5. Existing human behavioral datasets from experimental linguistics are reused as ready-made, already validated benchmarks with graded human expectations.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): crate prag could emit an evaluativity implicature for bare positive gradable predicates and withhold it for comparatives and equatives, with an explanation trace citing the construction type.
  • I2 (world): the measures/values lexicon could tag each gradable adjective with its class and scale, so the world model knows whether "X is full" fixes a state or "X is tall" only places X relative to a comparison class.
  • I3 (frames): implicature computation could take a speaker-profile parameter (strict/literal vs. loose/pragmatic) that adjusts precision standards for numerals and gradable terms.
  • I4 (learn): evaluation of inference outputs should include such distractor controls and score with explicit entailment over meaning graphs rather than surface similarity.
  • I5 (learn): Mova's pragmatics module could be checked against published psycholinguistic judgment patterns, comparing the direction of effects rather than single accuracy numbers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksImplicature · 138 works

Cite

Cite as Yan Cong. 2022. Pre-trained Language Models' Interpretation of Evaluativity Implicature: Evidence from Gradable Adjectives Usage in Context. Proceedings of the Second Workshop on Understanding Implicit and Underspecified Language.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.unimplicit-1.1

@inproceedings{cong2022pre,
  title = {{Pre-trained Language Models' Interpretation of Evaluativity Implicature: Evidence from Gradable Adjectives Usage in Context}},
  author = {Yan Cong},
  booktitle = {Proceedings of the Second Workshop on Understanding Implicit and Underspecified Language},
  year = {2022},
  doi = {10.18653/v1/2022.unimplicit-1.1},
  url = {https://doi.org/10.18653/v1/2022.unimplicit-1.1}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. William Labov. 2006. The social stratification of English in New York city. Cambridge University Press. ↗
  2. Andrea Beltrama and Florian Schwarz. 2021. Imprecision, personae, and pragmatic reasoning. In Semantics and Linguistic Theory, volume 31, pages 122–144. ↗
  3. Nancy Niedzielski. 1999. The effect of social information on the perception of sociolinguistic variables. Journal of language and social psychology, 18(1):62– 85. ↗
  4. Manfred Bierwisch. 1989. The semantics of gradation. bierwisch, manfred & ewald lang (eds.), dimensional adjectives. find
  5. Lalchand Pandia, Yan Cong, and Allyson Ettinger. 2021. Pragmatic competence of pre-trained language models through the lens of discourse connectives. arXiv preprint arXiv:2109.12951. ↗
  6. Adrian Brasoveanu and Jessica Rett. 2018. Evaluativity across adjective and construction types: An experimental study. Journal of Linguistics, 54(2):263–329. ↗
  7. Robert J Podesva. 2011. Salience and the social meaning of declarative contours: Three case studies of gay professionals. Journal of english linguistics, 39(3):233–264. ↗
  8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  9. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  10. Dylan Bumford and Jessica Rett. 2020. Rationalizing evaluativity. In Sinn und Bedeutung 25. find
  11. Laura Staum Casasanto. 2008. Does social information influence sentence processing? In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 30. find
  12. J. Rett. 2008a. Antonymy and evaluativity. Proceedings of SALT XVII. CLC Publications. In J. Rett. 2008b. Degree Modification in Natural Language. Ph.D. thesis, Rutgers University. ↗
  13. June Choe, Shayne Sloggett, Masaya Yoshida, and Annette D’onofrio. 2019. Personae in syntactic processing: Socially-specified agents bias expectations of verb transitivity. In Poster presented at the 32nd CUNY Conference on Human Sentence Processing. find
  14. Jessica Rett. 2015. The Semantics of Evaluativity. Oxford University Press. ↗
  15. Jessica Rett. 2019. Manner implicatures and how to spot them. International Review of Pragmatics, in press. ↗
  16. M. Cresswell. 1976. The semantics of degree. In B.H. Partee, editor, Montague Grammar, 261-292. Academic Press. ↗
  17. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. ↗
  18. Annette D’onofrio. 2018. Personae and phonetic detail in sociolinguistic signs. Language in Society, 47(4):513–539. ↗
  19. Penelope Eckert. 2008. Variation and the indexical field 1. Journal of sociolinguistics, 12(4):453–476. find
  20. Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020. Harnessing the linguistic signal to predict scalar inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5387–5403, Online. Association for Computational Linguistics. ↗
  21. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. ↗
  22. Dan Iter, Jong Yoon, and Dan Jurafsky. 2018. Automatic detection of incoherent speech for diagnosing schizophrenia. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, pages 136–146, New Orleans, LA. Association for Computational Linguistics. find
  23. Michael Silverstein. 2003. Indexical order and the dialectics of sociolinguistic life. Language & communication, 23(3-4):193–229. find
  24. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. ↗
  25. Oyvind Tafjord and Peter Clark. 2021. General-purpose question-answering with Macaw. ArXiv, abs/2109.02593. ↗
  26. Elizabeth A Strand. 1999. Uncovering the role of gender stereotypes in speech perception. Journal of language and social psychology, 18(1):86–100. ↗
  27. Arnim von Stechow. 1984. Comparing semantic theories of comparison. Journal of semantics, 3(3):1–77. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cong-2022-pre-trained-language-models-interpretation/