Elamur Research⌕
☰
home / papers / 2023 / Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models

Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models

Paper · 2023 · Yiyuan Li, Rakesh Menon, Sayan Ghosh et al.

Summary

Vague quantifiers like few, most or some imply a proportion, but foundation models never receive direct supervision on what share of cases each one means. The authors treat quantifier interpretation as predicting a percentage interval on a discretized 0–100% grid. Their framework, PRESQUE, has three parts:

  • Literal listener. An NLI model built on encoders such as BERT and RoBERTa scores how strongly a quantified sentence entails a variant that states a concrete percentage.
  • Literal speaker. The same entailment check, run with premise and hypothesis swapped.
  • Pragmatic listener. A Rational Speech Acts step that combines the speaker with a prior derived from word frequencies.

To test on realistic text, they built QuRe. It contains Wikipedia sentences whose percentage mentions were rewritten with quantifiers. Each item has a gold percentage scope and a specificity label, which says how much the surrounding context alone reveals the value. Without extra training, the pragmatic listener beats the literal baseline by roughly 20% span-based F1 at all specificity levels. Results on HVD point the same way. This suggests pragmatic inference can extract usable quantity meaning from off-the-shelf inference models.

Key points

  • Formula. PRESQUE computes L1(p|q) ∝ S0(q|p)·P(p).
    • S0 is the NLI entailment score with premise and hypothesis reversed.
    • The prior P(p) is a sum over quantifiers of P(p|q)P(q).
    • P(q) is taken from the WORDFREQ resource.
  • QuRe construction.
    • HVD ⟨concept, feature, quantifier⟩ triples were turned into template sentences.
    • These sentences were linked to related Wikipedia entities.
    • About 5,000 candidate sentences with percentages were extracted by regex.
    • Candidates were filtered to 10–40 space-separated tokens, and percentages describing comparative changes were excluded.
  • LLM filtering check. GPT-3.5-turbo flagged sentences with comparative changes. On a hand-labeled held-out set of 50 sentences it reached 0.76 F1.
  • Evaluation.
    • Metrics: HIT@1, MRR, span-based F1 against gold scopes, a distance-based scope metric, and human preference judgments collected via Amazon Mechanical Turk.
    • PRESQUE leads on the reported metrics.
    • It produces tighter scopes for no and all, and wider ones for vaguer quantifiers.
  • Limitations. The data covers only Wikipedia, English, and a small subset of quantifiers. Scope judgments are inherently subjective.

Interesting ideas

  • I1. Running the same entailment model in reverse yields a speaker model at no extra cost, and a Bayesian listener built on top of it recovers pragmatic readings that literal entailment misses.
  • I2. Representing a quantifier's meaning as a distribution over fixed proportion bins, rather than a single number or a symbolic label, keeps the vagueness explicit and comparable across words.
  • I3. The dataset is created backwards: real sentences with exact numbers have those numbers swapped for vague words, so the gold answer comes from the original text for free.
  • I4. Each item is tagged by whether the context alone fully, partially, or not at all determines the hidden value, which separates reading from relying on priors.
  • I5. Before trusting an LLM filter at scale, the authors measured it on a small hand-annotated held-out set.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag could derive scalar implicatures (e.g. some suggesting "not all") by checking entailment in both directions over frames and normalizing like an RSA listener, so every step stays inspectable.
  • I2 (world): quantified statements in the world of a text could be stored as proportion intervals over entity sets, which maths word problems and consistency checks can then reason with.
  • I3 (learn): self-play could replace numbers or explicit facts in corpus text with vague or indirect wording and use the original as a silver target for training and evaluation.
  • I4 (answer): fable and school-test QA items could carry a similar label, showing whether the readers answer from the text or from compiled background knowledge.
  • I5 (learn): each LLM teacher tier could be calibrated on a tiny gold sample before its labels enter silver data, with the measured F1 deciding how much weight it gets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of North Carolina at Chapel HillChapel Hill · United States67 works · 16,474 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Yiyuan Li, Rakesh Menon, Sayan Ghosh and Shashank Srivastava. 2023. Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.38 · arXiv 2311.04659

@inproceedings{li2023pragmatic,
  title = {{Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models}},
  author = {Yiyuan Li, Rakesh Menon, Sayan Ghosh and Shashank Srivastava},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.38},
  eprint = {2311.04659},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.38}
}

References

37 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rachid Alami. 2013. On human models for collaborative robots. In 2013 International Conference on Collaboration Technologies and Systems (CTS), pages 191–194. find
  2. Leon Bergen, R. Levy, and Noah D. Goodman. 2016. Pragmatic reasoning through semantic inference. Semantics and Pragmatics, 9. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R’e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021. On the opportunities and risks of foundation models. ArXiv, abs/2108.07258. ↗
  3. Jonathan Gordon and Lenhart K. Schubert. 2010. Quantificational sharpening of commonsense knowledge. In AAAI Fall Symposium: Commonsense Knowledge. find
  4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. ↗
  5. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
  6. Fausto Carcassi and Jakub Szymanik. 2021. ‘most’ vs ‘more than half’: An alternatives explanation. In Proceedings of the Society for Computation in Linguistics 2021, pages 334–343, Online. Association for Computational Linguistics. find
  7. Ruixiang Cui, Daniel Hershcovich, and Anders Søgaard. 2022. Generalized quantifiers as a source of error in multilingual NLU benchmarks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4875–4893, Seattle, United States. Association for Computational Linguistics. ↗
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  9. Michael C. Frank and Noah D. Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998. ↗
  10. Noah D. Goodman and Michael C. Frank. 2016. Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences, 20(11):818–829. ↗
  11. H. P. Grice. 1975. Logic and Conversation, pages 41 – 58. Brill, Leiden, The Netherlands. find
  12. Aurélie Herbelot and Eva Maria Vecchi. 2015. From concepts to models : Some issues in quantifying feature norms. In Linguistic Issues in Language Technology (LiLT). find
  13. Alexandra Horowitz, Rose M. Schneider, and Michael C. Frank. 2018. The trouble with quantifiers: Exploring children’s deficits in scalar implicature. Child development, 89 6:e572–e593. ↗
  14. Pratik Joshi, Somak Aditya, Aalok Sathe, and Monojit Choudhury. 2020. TaxiNLI: Taking a ride up the NLU hill. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 41–55, Online. Association for Computational Linguistics. find
  15. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. ↗
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. ↗
  17. Andrzej Wlodzimierz Mostowski. 1957. On a generalization of quantifiers. Studies in logic and the foundations of mathematics, 93:311–335. ↗
  18. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics. ↗
  19. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Association for the Advancement of Artificial Intelligence (AAAI). find
  20. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computational Linguistics. ↗
  21. OpenAI. 2022. Introducing chatgpt. find
  22. Sandro Pezzelle, Raffaella Bernardi, and Manuela Piazza. 2018a. Probing the mental representation of quantifiers. Cognition, 181:117–126. ↗
  23. Sandro Pezzelle, Ionut-Teodor Sorodoc, and Raffaella Bernardi. 2018b. Comparatives, quantifiers, proportions: a multi-task model for the learning of quantities from vision. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 419–430, New Orleans, Louisiana. Association for Computational Linguistics. find
  24. Steven T. Piantadosi, Harry J. Tily, and Edward Gibson. 2012. The communicative function of ambiguity in language. Cognition, 122:280–291. ↗
  25. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. ↗
  26. Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019. EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 349–361, Hong Kong, China. Association for Computational Linguistics.
  27. Kyle Richardson, Hai Hu, Lawrence Moss, and Ashish Sabharwal. 2020. Probing natural language inference models through semantic fragments. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8713–8721. ↗
  28. Kyle Richardson, Hai Hu, Lawrence S Moss, and Ashish Sabharwal. 2019. Probing Natural Language Inference Models through Semantic Fragments. In Association for the Advancement of Artificial Intelligence (AAAI). ↗
  29. Subhro Roy, Tim Vieira, and Dan Roth. 2015. Reasoning about quantities in natural language. Transactions of the Association for Computational Linguistics, 3:1–13. ↗
  30. Stephanie Solt. 2016. On measurement and quantification: The case of most and more than half. Language, 92:100 – 65. ↗
  31. Robyn Speer. 2022. rspeer/wordfreq: v3.0. find
  32. Shashank Srivastava, Igor Labutov, and Tom Mitchell. 2018. Zero-shot learning of classifiers from natural language quantification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 306–316. find
  33. Alberto Testoni, Sandro Pezzelle, and Raffaella Bernardi. 2019. Quantifiers in a multimodal world: Hallucinating vision with language and sound. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 105–116, Minneapolis, Minnesota. Association for Computational Linguistics. find
  34. Shivin Thukral, Kunal Kukreja, and Christian Kavouras. 2021. Probing language models for understanding of temporal expressions. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 396–406, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  35. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Transactions on Machine Learning Research. Survey Certification. ↗
  36. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. ↗
  37. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2023-pragmatic-reasoning-unlocks-quantifier-semantics/