Elamur Research⌕
☰
home / papers / 2024 / Scope Ambiguities in Large Language Models

Scope Ambiguities in Large Language Models

Paper · 2024 · Gaurav Kamath, Sebastian Schuster, Sowmya Vajjala et al.

Summary

The paper asks how autoregressive LLMs handle scope ambiguity. This occurs when a sentence contains several semantic operators, such as quantifiers, negation or adverbs like "usually", and their relative scope can be ordered in more than one way. Resolving such sentences depends on both semantic structure and world knowledge. The authors built new English datasets of nearly 1,000 distinct scope-ambiguous sentences, each annotated with human judgments, and tested GPT-2, GPT-3/3.5, Llama 2 and GPT-4. Experiment 1 checks whether models favour the same reading as humans, using a question-answering setup. Experiment 2 uses probabilities of acceptable versus unacceptable follow-up sentences to test whether models register that two readings exist. The stronger models matched human preferences with high accuracy, above 90% in some cases, and showed sensitivity to the ambiguity itself. These results run counter to earlier zero-shot studies. The authors argue that probability-based probing and metalinguistic prompting can lead to different conclusions about what LLMs know.

Key points

  • Data: Roughly 1,000 unique scope-ambiguous sentences with human annotations, which the authors describe as the largest resource of its kind. The sentences cover quantifier–quantifier, quantifier–negation and quantifier–adverb interactions. Human judgments were collected via Prolific. Data and code are available at github.com/McGill-NLP/scope-ambiguity.
  • Experiment 2B (110 items): text-davinci-003 and Llama2-13b produced positive α-scores, indicating a preference for acceptable follow-ups, on more than 90% of items. Llama 2 chat variants correlated better with human proxy scores than the base models.
  • Reformatted test: The authors recast part of their own data into the True/False format of Liu et al. (2023). Most models then performed near chance, and even GPT-4 and GPT-3.5-turbo reached only 64%.
  • Earlier datasets: A manual review of the 579 ambiguous sentences in Liu et al.'s dataset found only about 20 genuine scope ambiguities.
  • Limitations: The study covers English only and does not examine how context shifts the preferred reading.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️National Research Council CanadaOttawa · Canada59 works · 21,677 citations🎓University College LondonLondon · United Kingdom88 works · 19,432 citations🎓McGill UniversityMontreal · Canada87 works · 11,027 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada54 works · 13,565 citations🔬Canadian Institute for Advanced ResearchToronto · Canada28 works · 16,725 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Gaurav Kamath, Sebastian Schuster, Sowmya Vajjala and Siva Reddy. 2024. Scope Ambiguities in Large Language Models. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00670

@article{kamath2024scope,
  title = {{Scope Ambiguities in Large Language Models}},
  author = {Gaurav Kamath, Sebastian Schuster, Sowmya Vajjala and Siva Reddy},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00670},
  url = {https://doi.org/10.1162/tacl_a_00670}
}

References

3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. a. Context: Ada finds it hard to study alone, so she generally invites others for joint study sessions. find
  2. Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022. A review on language models as knowledge bases. arXiv preprint arXiv:2204.06031. b. Context: Ada, Rohan, and Jo are good friends in the same program, and prepare for exams together. find
  3. Scott AnderBois, Adrian Brasoveanu, and Robert Henderson. 2012. The pragmatics of quantifier scope: A corpus study. In Proceedings of Sinn und Bedeutung, volume 16, pages 15–28. find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kamath-2024-scope-ambiguities-large-language-models/