Elamur Research⌕
☰
home / papers / 2023 / Zero and Few-shot Semantic Parsing with Ambiguous Inputs

Zero and Few-shot Semantic Parsing with Ambiguous Inputs

Paper · 2023 · Elias Stengel-Eskin, Kyle Rawlins and Benjamin Van Durme

Summary

Semantic parsing benchmarks usually assume each utterance has exactly one correct formal meaning, so it has been unclear whether parsers built on large language models represent ambiguity at all. Stengel-Eskin, Rawlins and Van Durme address this with AMP (Ambiguous Parsing), a template-based framework, dataset and challenge. In AMP, each ambiguous sentence is paired with two logical forms, written in first-order logic (FOL) or Lisp. The templates cover five classic ambiguity types and also generate unambiguous controls.

The authors use in-context learning to test several few-shot text-to-code models in two settings:

  • Zero-shot: the prompt supplies the building blocks for both readings but no example of the ambiguity itself.
  • Mixed-prompt: the prompt contains conflicting examples for each reading in varying proportions.

They propose three new metrics and compare model outputs with crowdsourced human judgements. Models mostly fail at zero-shot composition and seldom produce both parses. However, some closely follow the proportions of readings shown in conflicting prompts. The authors conclude that datasets should explicitly include ambiguity and that evaluation should consider distributions over outputs.

Key points

  • AMP covers five ambiguity types: prepositional-phrase (PP) attachment, quantifier scope, inverse scope, pronoun coreference (bound), and conjunctions. Logical forms are canonicalized by ordering binary trees and anonymizing variables, so logically equivalent forms count as identical.
  • Locally run models use grammar-constrained decoding via BenchCLAMP with a beam size of 5. Every output is therefore well-formed, which separates semantic errors from syntactic ones.
  • An error analysis of Codegen-16B sampled 10 wrong outputs per ambiguity type:
    • Inverse scope: all 10 put ∃ where ∀ was needed.
    • PP attachment: 9 of 10 omitted a predicate.
    • Bound: 8 of 10 reused one event variable for two verbs.
  • Human judgements came from Mechanical Turk annotators paid about $14/hour. Each did 20 items and rated confidence on a slider, which was mapped to probabilities between 0.5 and 1.0. Visual verb–noun pairs (e.g., saw–telescope) favoured the instrument reading, while tactile ones (e.g., held–gloves) favoured possession.
  • According to the authors, no earlier work had examined how in-context learning behaves when prompts contain contradictory evidence. Data and code are released at github.com/esteng/ambiguous_parsing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations🎓University of North Carolina at Chapel HillChapel Hill · United States67 works · 16,474 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Semantic parsing & meaning representation · 185 works

Cite

Cite as Elias Stengel-Eskin, Kyle Rawlins and Benjamin Van Durme. 2023. Zero and Few-shot Semantic Parsing with Ambiguous Inputs. arXiv:2306.00824.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.00824 · arXiv 2306.00824

@misc{stengeleskin2023zero,
  title = {{Zero and Few-shot Semantic Parsing with Ambiguous Inputs}},
  author = {Elias Stengel-Eskin, Kyle Rawlins and Benjamin {Van Durme}},
  year = {2023},
  doi = {10.48550/arxiv.2306.00824},
  eprint = {2306.00824},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.00824}
}

References

64 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Scott AnderBois, Adrian Brasoveanu, and Robert Henderson. The pragmatics of quantifier scope: A corpus study. In Proceedings of Sinn und Bedeutung, volume 16, pp. 15–28, 2012. find
  2. Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49–62, 2013.
  3. Yoav Artzi, Kenton Lee, and Luke Zettlemoyer. Broad-coverage ccg semantic parsing with amr. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1699–1710, 2015.
  4. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1533–1544, 2013.
  5. Yevgeni Berzak, Andrei Barbu, Daniel Harari, Boris Katz, and Shimon Ullman. Do you see what I mean? Visual resolution of linguistic ambiguities. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1477–1487, 2015. find
  6. Nilavra Bhattacharya, Qing Li, and Danna Gurari. Why does a visual question have different answers? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4271–4280, 2019. find
  7. Johan Bos. Computational semantics in discourse: Underspecification, resolution, and inference. Journal of Logic, Language and Information, 13:139–157, 2004. find
  8. Glenn W Brier et al. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1–3, 1950. find
  9. Alfonso Caramazza, Ellen Grober, Catherine Garvey, and Jack Yates. Comprehension of anaphoric pronouns. Journal of verbal learning and verbal behavior, 16(5):601–609, 1977. find
  10. Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. Uncertain natural language inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8772–8779, Online, July 2020. Association for Computational Linguistics. doi: 1…
  11. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna… ↗
  12. Ann Copestake, Dan Flickinger, Carl Pollard, and Ivan A Sag. Minimal recursion semantics: An introduction. Research on language and computation, 3:281–332, 2005. find
  13. Marco Damonte, Rahul Goel, and Tagyoung Chung. Practical semantic parsing for spoken language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pp. 16–23, 2019. find
  14. Li Dong and Mirella Lapata. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 33–43, Berlin, Germany, 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1004. URL…
  15. Veena D Dwivedi. Interpreting quantifier scope ambiguity: Evidence of heuristic first, algorithmic second processing. PloS one, 8(11):e81461, 2013. find
  16. Ruth Filik, Kevin B Paterson, and Simon P Liversedge. Processing doubly quantified sentences: Evidence from eye movements. Psychonomic Bulletin & Review, 11(5):953–959, 2004. find
  17. Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3–71, 1988. find
  18. Matthieu Futeral, Cordelia Schmid, Ivan Laptev, Benoı̂t Sagot, and Rachel Bawden. Tackling ambiguity with images: Improved multimodal machine translation and contrastive evaluation. arXiv preprint arXiv:2212.10140, 2022. find
  19. Rohit J Kate, Yuk Wah Wong, Raymond J Mooney, et al. Learning to transform natural to formal languages. In AAAI, volume 5, pp. 1062–1068, 2005. find
  20. Najoung Kim and Tal Linzen. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9087–9105, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/202… find
  21. Howard S Kurtzman and Maryellen C MacDonald. Resolution of quantifier scope ambiguities. Cognition, 48(3):243–279, 1993. find
  22. James R Lackner and Merrill F Garrett. Resolving ambiguity: Effects of biasing context in the unattended ear. Cognition, 1(4):359–372, 1972. find
  23. Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock-holmsmässan, Stockholm,… find
  24. PMLR, 2018. URL http://proceedings.mlr.press/v80/ lake18a.html . ↗
  25. Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. We’re afraid language models aren’t modeling ambiguity. arXiv preprint arXiv:2304.14399, 2023.
  26. Ninareh Mehrabi, Palash Goyal, Apurv Verma, Jwala Dhamala, Varun Kumar, Qian Hu, Kai-Wei Chang, Richard Zemel, Aram Galstyan, and Rahul Gupta. Is the elephant flying? resolving ambiguities in text-to-image generative models. arXiv preprint arXiv:2211.12503, 2022. find
  27. Grégoire Mialon, Roberto Dessı̀, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842, 2023.
  28. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5783–5797, 2020. find
  29. Richard Montague. English as a formal language. Linguaggi nella Societa e nella Tecnica, 1970. find
  30. Yixin Nie, Xiang Zhou, and Mohit Bansal. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020.
  31. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474, 2022. find
  32. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744, 2022.
  33. Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255, 2022.
  34. Terence Parsons. Events in the Semantics of English: A Study in Subatomic Semantics. MIT Press, 1990. find
  35. Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694, 2019.
  36. Steven T Piantadosi, Harry Tily, and Edward Gibson. The communicative function of ambiguity in language. Cognition, 122(3):280–291, 2012. find
  37. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners, 2019. find
  38. Nathan Ellis Rasmussen and William Schuler. A corpus of encyclopedia articles with logical forms. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 2020. find
  39. Keith Rayner, Marcia Carlson, and Lyn Frazier. The interaction of syntax and semantics during sentence processing: Eye movements in the analysis of semantically biased sentences. Journal of verbal learning and verbal behavior, 22(3):358–374, 1983. find
  40. Subhro Roy, Sam Thomson, Tongfei Chen, Richard Shin, Adam Pauls, Jason Eisner, and Benjamin Van Durme. BenchCLAMP: A benchmark for evaluating language models on semantic parsing. arXiv preprint arXiv:2206.10668, 2022. — [arXiv:2206.10668](https://arxiv.org/abs/2206.10668) find
  41. Rachel Rudinger and Benjamin Van Durme. Is the stanford dependency representation semantic? In Proceedings of the Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pp. 54–58, 2014. find
  42. Keisuke Sakaguchi and Benjamin Van Durme. Efficient online scalar annotation with bounded support. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 208–218, Melbourne, Australia, July 2018. Association for Computational Linguisti… find
  43. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
  44. Hinrich Schutze. Ambiguity in language learning: Computational and cognitive models. Stanford University, 1995. find
  45. Semantic Machines, Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman,… — Task-Oriented Dialogue as Dataflow Synthesis
  46. Richard Shin and Benjamin Van Durme. Few-shot semantic parsing with language models trained on code. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5417–5425, 2022.
  47. Richard Shin, Christopher Lin, Sam Thomson, Charles Chen Jr, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. Constrained language models yield few-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Langu… find
  48. Mark Steedman. Taking scope: The natural semantics of quantifiers. Mit Press, 2011. find
  49. Elias Stengel-Eskin and Benjamin Van Durme. Calibrated interpretation: Confidence estimation in semantic parsing. arXiv preprint arXiv:2211.07443, 2022. find
  50. Elias Stengel-Eskin and Benjamin Van Durme. Did you mean...? Confidence-based trade-offs in semantic parsing. arXiv preprint arXiv:2303.16857, 2023. find
  51. Elias Stengel-Eskin, Jimena Guallar-Blasco, Yi Zhou, and Benjamin Van Durme. Why did the chicken cross the road? Rephrasing and analyzing ambiguous questions in vqa. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023. find
  52. Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pp. 1507–1514, 2011. find
  53. Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3:25–55, 2020. find
  54. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  55. Siddharth Vashishtha, Benjamin Van Durme, and Aaron Steven White. Fine-grained temporal relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2906–2919, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P… find
  56. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
  57. Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1–191, 1972. find
  58. Ludwig Wittgenstein. Tractatus Logico-Philosophicus. Annalen der Naturphilosophie, 1921. find
  59. Zhengxuan Wu, Christopher D Manning, and Christopher Potts. ReCOGS: How incidental details of a logical form overshadow an evaluation of semantic interpretation. arXiv preprint arXiv:2303.13716, 2023. find
  60. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods…
  61. Yuewei Yuan, Chaitanya Malaviya, and Mark Yatskar. AmbiCoref: Evaluating human and model sensitivity to ambiguous coreference. In Findings of the Association for Computational Linguistics: EACL 2023, pp. 993–1000, 2023.
  62. John M Zelle and Raymond J Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pp. 1050–1055, 1996.
  63. Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. Broad-coverage semantic parsing as transduction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3786–3798…
  64. George Kingsley Zipf. Human Behaviour and the Principle of Least Effort. Addison-Wesley Press, 1949. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stengel-eskin-2023-zero-few-shot-semantic-parsing/