Elamur Research⌕
☰
home / papers / 2022 / Predicting scalar diversity with context-driven uncertainty over alternatives

Predicting scalar diversity with context-driven uncertainty over alternatives

Paper · 2022 · Jennifer Hu, Roger Levy and Sebastian Schuster

Summary

Scalar implicatures, such as reading "some" as "not all," are drawn at very different rates depending on the scale involved. The authors ask whether this scalar diversity can be explained by how sure a listener is about which scale the speaker has in mind. They use the adjectival items (32 scales) from van Tiel et al.'s (2016) Experiment 2. Each weak adjective is placed in a frame like "[NP] is [WEAK], but not ___." A T5 model fine-tuned for text infilling then fills the gap, giving either the probability of the tested strong scalemate or a set of sampled alternatives. Uncertainty is measured as entropy in two ways:

  • over the sampled word strings;
  • over latent clusters of alternatives, found with Gaussian mixtures over Sentence-T5 embeddings.

Both entropy measures predict human SI rates, with lower uncertainty linked to more implicatures. The surprisal of the strong scalemate predicts nothing, whether computed on strings or clusters. This shifts the notion of availability away from a single competitor word and toward the spread of plausible alternatives in context. It also gives a way to test concept-level theories of alternatives quantitatively.

Key points

  • Human data: SI rates come from van Tiel et al. (2016), Experiment 2, adjectival scales only (32 ⟨WEAK, STRONG⟩ pairs, 3 NP contexts each).
  • Model: T5-base from Huggingface Transformers, fine-tuned on span infilling over roughly 10M tokens of the 2007 New York Times Corpus.
  • Sampling: 100 completions per template, capped at 5 tokens. Outputs are reduced to their first word, with punctuation, casing, duplicates and stopword-only completions removed.
  • Results: All regressions include scale boundedness as a control, and boundedness stays highly significant.
    • String entropy: t = −3.28, p = 0.001 (significant).
    • Class entropy: t = −2.4, p = 0.01 (significant).
    • String surprisal: t = −0.09, p = 0.928 (not significant).
    • Class surprisal: t = −1.33, p = 0.186 (not significant).
  • Clustering: Alternatives are embedded as full sentences with Sentence-T5 (768-dim). They are grouped with a BIC-selected GMM using diagonal covariances. For "The salary is adequate, but not ___," the model finds an employee-side cluster (good, generous) and an employer-side cluster (extravagant).

Interesting ideas

  • I1. A contrastive frame like "X, but not ___" pushes a gap-filling model to propose scalemates of X, so the competing alternatives can be read straight off the model's output.
  • I2. The paper models how sure the listener is about the scale as entropy over the whole set of alternatives, rather than scoring one particular competitor; this predicts behaviour where the single-competitor score fails.
  • I3. Alternatives are grouped into concepts by embedding the completed sentence, not the bare word, and clustering with a BIC-selected mixture, so synonyms count as one alternative.
  • I4. One weak word can belong to several scales depending on perspective (an adequate salary from the employee's versus the employer's view), and the clusters make this ambiguity visible.
  • I5. The authors caution that asking participants about the strong term directly may hide retrieval effects, a reminder that the test format can mask the variable being studied.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The implicature module could probe a weak term with such a template to build a context-specific list of stronger alternatives, instead of relying on a fixed scale table.
  • I2 (frames): The prag crate could assign each implicature a strength that falls as alternative entropy rises, and log that entropy as the explanation for a weak or cancelled inference.
  • I3 (world): When compiling scales or value orderings into knowledge, the system could cluster alternatives by meaning before building the scale, so near-synonyms do not split probability mass.
  • I4 (world): The text world could keep more than one candidate scale for an evaluative adjective, tied to whose viewpoint holds, until context selects one.
  • I5 (learn): Evaluations of implicature could include versions that do not name the alternative, as a control against format-driven success.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States136 works · 86,390 citations🎓Massachusetts Institute of TechnologyCambridge · United States159 works · 65,157 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works

Cite

Cite as Jennifer Hu, Roger Levy and Sebastian Schuster. 2022. Predicting scalar diversity with context-driven uncertainty over alternatives. Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.cmcl-1.8

@inproceedings{hu2022predicting,
  title = {{Predicting scalar diversity with context-driven uncertainty over alternatives}},
  author = {Jennifer Hu, Roger Levy and Sebastian Schuster},
  booktitle = {Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.cmcl-1.8},
  url = {https://doi.org/10.18653/v1/2022.cmcl-1.8}
}

References

28 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brian Buccola, Manuel Križ, and Emmanuel Chemla. 2021. Conceptual alternatives: Competition in language and beyond. Linguistics and Philosophy. find
  2. Elizabeth Pankratz and Bob van Tiel. 2021. The role of relevance for scalar diversity: a usage-based approach. Language and Cognition, 13(4):562–594. Edition: 2021/08/16 Publisher: Cambridge University Press. ↗
  3. Gerard de Melo and Mohit Bansal. 2013. Good, Great, Excellent: Global Inference of Semantic Intensities. Transactions of the Association for Computational Linguistics, 1:279–290. ↗
  4. Peng Qian and Roger Levy. 2022. Flexible generation from fragmentary linguistic input. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. To appear. ↗
  5. Judith Degen. 2015. Investigating the distribution of some (but not all) implicatures using corpora and web-based methods. Semantics and Pragmatics, 8(11):1–55. ↗
  6. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  7. Ryan Doran, Rachel E. Baker, Yaron McNabb, Meredith Larson, and Gregory Ward. 2009. On the Non-Unified Nature of Scalar Implicature: An Empirical Investigation. International Review of Pragmatics, 1(2):211 – 248. Place: Leiden, The Netherlands Publisher: Brill. ↗
  8. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  9. Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah. Association for Computational Linguistics.
  10. Eszter Ronai and Ming Xiang. 2022. Three factors in explaining scalar diversity. In Proceedings of Sinn und Bedeutung 26. ↗
  11. Evan Sandhaus. 2008. The New York Times Annotated Corpus (LDC2008T19). find
  12. Nicole Gotzner, Stephanie Solt, and Anton Benz. 2018. Scalar Diversity, Negative Strengthening, and Adjectival Semantics. Frontiers in Psychology, 9:1659. ↗
  13. Suzanna Sia, Ayush Dalmia, and Sabrina J. Mielke. 2020. Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too! In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1728–1736, Online. Association for Computational Linguistics. find
  14. Herbert P. Grice. 1975. Logic and Conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Semantics: Speech Acts, volume 3, pages 41–58. Academic Press. ↗
  15. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302 – 319. ↗
  16. Marti A. Hearst. 1992. Automatic Acquisition of Hyponyms from Large Text Corpora. In COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics. ↗
  17. Dan Sperber and Deirdre Wilson. 1986. Relevance: Communication and Cognition. Wiley-Blackwell. find
  18. Julia Bell Hirschberg. 1985. A Theory of Scalar Implicature (Natural Languages, Pragmatics, Inference). PhD Thesis, University of Pennsylvania. find
  19. Chao Sun, Ye Tian, and Richard Breheny. 2018. A Link Between Local Enrichment and Scalar Diversity. Frontiers in Psychology, 9:2092. ↗
  20. Elissa Li, Sebastian Schuster, and Judith Degen. 2021. Predicting Scalar Inferences From "Or" to "Not Both" Using Neural Sentence Encoders. In Proceedings of the Society for Computation in Linguistics, volume 4. ↗
  21. Laure Thompson and David Mimno. 2020. Topic Modeling with Contextualized Word Representation Clusters. ↗
  22. Yu Meng, Yunyi Zhang, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations. In The Web Conference. find
  23. Bob van Tiel, Emiel van Miltenburg, Natalia Zevakhina, and Bart Geurts. 2016. Scalar Diversity. Journal of Semantics, 33(1):137–175. find
  24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  25. Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. 2021. Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models.
  26. Matthijs Westera and Gemma Boleda. 2020. A closer look at scalar diversity using contextualized semantic similarity. Proceedings of Sinn und Bedeutung, 24(2):439–454. ↗
  27. Ethan Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. In Proceedings of the Cognitive Science Society. find
  28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hu-2022-predicting-scalar-diversity-context-driven/