Elamur Research⌕
☰
home / papers / 2025 / Can LLMs Disambiguate Grounded Language? The Case of PP Attachment

Can LLMs Disambiguate Grounded Language? The Case of PP Attachment

Paper · 2025 · John Blackmore and Matthew Stone

Summary

The paper asks whether current LLMs, used zero-shot, can choose where a second prepositional phrase attaches in English phrases of the shape X p1 Y p2 Z: to the first noun X or to the nearer noun Y. The authors mined FlickR 30k image captions with spaCy tagging and chunking and found 730 candidate 5-tuples. They hand-labelled each as X, Y, or "doesn't matter", and noted whether the photo was needed to decide. Most cases (532) turned out to have no effect on grounded meaning.

On the remaining cases they compared two kinds of signal:

  • Plausibility: model probabilities for paraphrases that spell out each attachment.
  • Information structure: additional probabilities meant to capture foregrounding, fluency and word-order conventions.

These signals were combined with linear classifiers under 5-fold cross-validation. Plausibility from Llama 3.1 alone reached 89.4%, and adding information structure raised it to 92.9%. The calibrated plausibility model VERA performed comparably. The work matters because it separates world knowledge from pragmatic packaging as distinct cues for resolving attachment.

Key points

  • Data: FlickR 30k (over 30,000 outdoor photos, more than 150,000 crowdsourced captions). Mining with spaCy plus manual checks yielded 730 NP+PP+PP tuples. Data and code are at github.com/depthfirst/pp-ambiguity.
  • Spurious ambiguity: 532 of the 730 tuples were excluded because the attachment choice did not change grounded meaning.
    • 322 were cases where the spatial relation carries over (e.g. a person on a bicycle at a place).
    • 138 were cases where X is a group or collective noun followed by "of".
  • Accuracy: Llama 3.1 plausibility gave 89.4%, and plausibility plus information structure gave 92.9%. VERA-T5 7B reached similar accuracy. VERA is a statement-plausibility scorer trained on 7M true and false statements drawn from 19 QA datasets and 2 KBs, and it is calibrated against reporting bias.
  • Pattern slices: accuracy was reported for third-person pronouns, p2 = "of", and "in/on the fore/background". With Llama, the "of" slice rose from 76% to 97% and the background slice from 56% to 78% once information structure was added.
  • Caveats: labels were assigned by the authors, and the sample is small. Remaining errors were classified as needing visual context, better plausibility, better information structure, idioms, or near-boundary scores.

Interesting ideas

  • I1. Each candidate attachment is turned into its own explicit paraphrase, and a language model scores which version reads as more likely, so the tree decision becomes an inspectable comparison of two sentences.
  • I2. The paper adds a third label, "doesn't matter", for cases where both attachments yield the same situation, such as transitive location relations or of-collectives.
  • I3. Plausibility (what tends to happen in the world) is kept apart from information structure (what is foregrounded, plus fluency conventions such as "of" coming before "in"), and the two are treated as independent cues.
  • I4. A calibrated plausibility scorer like VERA is used to counter reporting bias, since obvious facts are rarely written down and raw text probability favours odd events.
  • I5. Accuracy is broken down by lexical pattern, and the authors propose minimal pairs and pronoun-substitution probes to test whether a model is systematic.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse or the perceptron parser could pass low-margin PP heads to such a reranker, and the two paraphrases with their scores could be logged as the explanation.
  • I2 (world): the world builder could detect when both readings produce identical entity locations and stop treating the choice as an error, which would also make attachment-error metrics fairer.
  • I3 (frames): prag could hold explicit information-structure rules, such as a background PP attaching to the nearest NP or "of" preceding "in", as separate explainable features rather than folding them into one score.
  • I4 (learn): when an LLM teacher tier judges attachment or role plausibility, scores could be calibrated or cross-checked against a verifier to avoid systematically favouring unusual readings.
  • I5 (learn): the evaluation suite could include pattern-sliced PP-attachment minimal pairs as negative controls, exposing repair rules that succeed only on frequent phrasings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Rutgers, The State University of New JerseyNew Brunswick · United States36 works · 12,026 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 works

Cite

Cite as John Blackmore and Matthew Stone. 2025. Can LLMs Disambiguate Grounded Language? The Case of PP Attachment. Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI Era.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{blackmore2025can,
  title = {{Can LLMs Disambiguate Grounded Language? The Case of PP Attachment}},
  author = {John Blackmore and Matthew Stone},
  booktitle = {Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI Era},
  year = {2025},
  doi = {10.26615/978-954-452-098-4-020},
  url = {https://doi.org/10.26615/978-954-452-098-4-020}
}

References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James Allen, Hannah An, Ritwik Bose, Will de Beaumont, and Choh Man Teng. 2020. A broad-coverage deep semantic lexicon for verbs. ↗
  2. Yonatan Belinkov, Tao Lei, Regina Barzilay, and Amir Globerson. 2014. Exploring compositional architectures and word vector representations for prepositional phrase attachment. Transactions of the Association for Computational Linguistics, 2:561–572. ↗
  3. Emily M Bender and Alexander Koller. 2020. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5185–5198. ↗
  4. Eric Brill and Philip Resnik. 1994. A rule-based approach to prepositional phrase attachment disambiguation. arXiv preprint cmp-lg/9410026. find
  5. Michael Collins and James Brooks. 1995. Prepositional phrase attachment through a backed-off model. arXiv preprint cmp-lg/9506021.
  6. Pradeep Dasigi, Waleed Ammar, Chris Dyer, and Eduard Hovy. 2017. Ontology-aware token embeddings for prepositional phrase attachment. arXiv preprint arXiv:1705.02925. ↗
  7. C. Féry and S. Ishihara. 2016. The Oxford Handbook of Information Structure. Oxford Handbooks Series. Oxford University Press. find
  8. Henry Watson Fowle and Robert William Burchfield. 2000. The new Fowler’s modern English usage. Oxford University Press. find
  9. Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 workshop on Automated knowledge base construction, pages 25–30. find
  10. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. ↗
  11. Manfred Krifka. 2008. Basic notions of information structure. Acta Linguistica Hungarica, 55(3-4):243– 276. ↗
  12. Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. 2023. Vera: A general-purpose plausibility estimation model for commonsense statements. arXiv preprint arXiv:2305.03695. ↗
  13. Tom McArthur, Jacqueline Lam-McArthur, and Lise Fontaine. 2018. Oxford companion to the English language. Oxford University Press. find
  14. Ndapandula Nakashole and Tom Mitchell. 2015. A knowledge-intensive model for prepositional phrase attachment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 365–375. find
  15. Adwait Ratnaparkhi, Jeff Reynar, and Salim Roukos. 1994. A maximum entropy model for prepositional phrase attachment. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
  16. Richard Socher, John Bauer, Christopher D Manning, and Andrew Y Ng. 2013. Parsing with compositional vector grammars. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 455–465. find
  17. Stefanie A Tellex, Thomas Fleming Kollar, Steven R Dickerson, Matthew R Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 1507–1514. find
  18. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. ↗
  19. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. ↗
  20. Houquan Zhou, Yang Hou, Zhenghua Li, Xuebin Wang, Zhefeng Wang, Xinyu Duan, and Min Zhang. 2023. How well do large language models understand syntax? an evaluation by asking natural language questions. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/blackmore-2025-can-llms-disambiguate-grounded-language/