Elamur Research⌕
☰
home / papers / 2024 / Tree Transformer's Disambiguation Ability of Prepositional Phrase Attachment and Garden Path Effects

Tree Transformer's Disambiguation Ability of Prepositional Phrase Attachment and Garden Path Effects

Paper · 2024 · Lingling Zhou, Suzan Verberne and Gijs Wijnholds

Summary

The paper asks whether a Transformer that learns syntax without supervision can resolve two kinds of syntactic ambiguity: prepositional phrase (PP) attachment, which is structural, and garden path sentences, which are incremental. The authors train the Tree Transformer (Wang et al., 2019) from scratch on the Penn Treebank. This model induces constituency trees through a constrained attention layer while it learns a language modelling objective.

For PP attachment, they build a new test set by turning the quadruples of the Ratnaparkhi et al. (1994) corpus into full, natural-sounding sentences with BERT. They then check where the induced trees attach the PP. For garden paths, they measure surprisal at the disambiguating word on the Futrell et al. (2019) test suites.

The comparison systems are a supervised BiLSTM that does constituency parsing as sequence labelling, and an LLM prompting baseline. The Tree Transformer does show garden path effects. However, it attaches PPs much worse than the BiLSTM and responds less to lexical cues than large LSTMs. The authors conclude that supervised pre-Transformer parsers remain the safer option when input is ambiguous.

Key points

  • New PP test set: 1424 sentences, built only from WSJ-test items of the PP attachment corpus so that none overlap with PTB training data.
    • bert-base-uncased fills masked slots for a subject pronoun and modifiers of both nouns, turning each 4-tuple into a 7-tuple.
    • The NLTK POS tagger checks that each filled word has the right category.
    • The original noun/verb attachment labels are kept.
  • Garden path suites (Futrell et al., 2019): MV/RR has 28 items, and NP/Z (Overt Object) and NP/Z (Verb Transitivity) have 24 items each. Every item comes in 4 variants.
  • PP attachment errors: the Tree Transformer often fails to recognise the PP as a unit and leans toward verb attachment.
    • Both models score lower when the verb is functional (e.g. "be", "have") rather than notional.
    • This split points to reliance on lexical cues.
  • Garden path size and cues: the Tree Transformer's effects are about as small as TinyLSTM's and well below those of large LSTMs and RNNG.
    • Like those stronger models, it reacts to unambiguous participle forms.
    • Its sensitivity to verb transitivity falls just below the large LSTMs.
    • It reacts much more to an overt object than to transitivity.
  • Code and data: released at github.com/L-innnng/Tree-Transformer-analysis.

Interesting ideas

  • I1. Bare lexical tuples with gold labels can be grown into fluent test sentences by letting a masked LM fill fixed slots under POS constraints, which keeps the labels valid while making the inputs look like real text.
  • I2. The source items come only from the treebank's test portion, which guarantees the generated probes cannot leak into parser training data.
  • I3. Instead of judging the whole parse, the evaluation checks a single decision (where the PP attaches) and reports precision/recall/F1 separately for functional and notional verbs, which exposes lexical biases that a global score would hide.
  • I4. Garden path sensitivity is measured with a 2×2 design (ambiguous vs. disambiguated, crossed with a cue such as verb form or an overt object), and effect size is the extra surprisal at the disambiguating word.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): we could expand compact attachment or role tuples into natural sentences to probe our parsers without hand-writing every example.
  • I2 (learn): any synthetic evaluation set we derive from UD treebanks should trace back to held-out splits, giving a clean negative control against memorisation.
  • I3 (tree): we could report PP attachment accuracy for perceptron and gparse broken down by head-verb class, such as light versus content verbs, to locate systematic misattachments.
  • I4 (tree): an analogous minimal-pair suite could test whether our incremental parser's scores drop at the disambiguator and whether specific cues like commas or participles reduce the drop.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Leiden UniversityLeiden · Netherlands32 works · 1,074 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Psycholinguistics & language acquisition · 102 works

Cite

Cite as Lingling Zhou, Suzan Verberne and Gijs Wijnholds. 2024. Tree Transformer's Disambiguation Ability of Prepositional Phrase Attachment and Garden Path Effects. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.664

@inproceedings{zhou2024tree,
  title = {{Tree Transformer's Disambiguation Ability of Prepositional Phrase Attachment and Garden Path Effects}},
  author = {Lingling Zhou, Suzan Verberne and Gijs Wijnholds},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.664},
  url = {https://doi.org/10.18653/v1/2024.acl-long.664}
}

References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Thomas G Bever. 1970. The cognitive basis for linguistic structures. Cognition and the development of language. find
  2. Michael Collins and James Brooks. 1995. Prepositional phrase attachment through a backed-off model. In Third Workshop on Very Large Corpora.
  3. Vera Demberg and Frank Keller. 2008. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2):193–210. ↗
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  5. Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum. 2019. Unsupervised latent tree induction with deep inside-outside recursive auto-encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1129–1141, Minneapolis, Minnesota. Association for Computational Linguistics. find
  6. Christiane Fellbaum. 1998. WordNet: An electronic lexical database. MIT press. find
  7. Stefan L Frank and Rens Bod. 2011. Insensitivity of the human sentence-processing system to hierarchical structure. Psychological science, 22(6):829–834. ↗
  8. L FRAZIER. 1978. On comprehending sentences: Syntactic parsing strategies. Doctoral dissertation, University of Connecticut. find
  9. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 32–42, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  10. Carlos Gómez-Rodríguez and David Vilares. 2018. Constituent parsing as sequence labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1314–1324, Brussels, Belgium. Association for Computational Linguistics. find
  11. Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah. Association for Computational Linguistics.
  12. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics.
  13. Donald Hindle and Mats Rooth. 1993. Structural ambiguity and lexical relations. Computational Linguistics, 19(1):103–120.
  14. Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gábor Melis. 2019. Unsupervised recurrent neural network grammars. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1105–1117, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  15. John Kimball. 1973. Seven principles of surface structure parsing in natural language. Cognition, 2(1):15– 47. ↗
  16. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. ↗
  17. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Annalen der Physik, 19(2):313–330. ↗
  18. Michael EJ Masson and Geoffrey R Loftus. 2003. Using confidence intervals for graphically based data interpretation. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale, 57(3):203. ↗
  19. Adwait Ratnaparkhi. 1998. Statistical models for unsupervised prepositional phrase attachment. In COL-ING 1998 Volume 2: The 17th International Conference on Computational Linguistics. ↗
  20. Adwait Ratnaparkhi, Jeff Reynar, and Salim Roukos. 1994. A maximum entropy model for prepositional phrase attachment. In Proceedings of the Workshop on Human Language Technology, HLT ’94, page 250–255.
  21. Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojević, Phil Blunsom, and Chris Dyer. 2022. Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale. Transactions of the Association for Computational Linguistics, 10:1423–1439. ↗
  22. Yikang Shen, Zhouhan Lin, Chin-Wei Huang, and Aaron C. Courville. 2017. Neural language modeling by jointly learning syntax and lexicon. ArXiv, abs/1711.02013. ↗
  23. Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron C. Courville. 2018. Ordered neurons: Integrating tree structures into recurrent neural networks. ArXiv, abs/1810.09536. ↗
  24. Jiri Stetina and Makoto Nagao. 1997. Corpus based PP attachment ambiguity resolution with a semantic dictionary. In Fifth Workshop on Very Large Corpora.
  25. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1556– 1566, Beijing, China. Association for Computational Linguistics. find
  26. Kristina Toutanova, Christopher D. Manning, and Andrew Y. Ng. 2004. Learning random walk models for inducing word dependency distributions. In Proceedings of the Twenty-First International Conference on Machine Learning, ICML ’04, page 103, New York, NY, USA. Association for Computing Machinery. find
  27. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023. Zephyr: Direct distillation of lm alignment. ↗
  28. Marten van Schijndel and Tal Linzen. 2018a. Modeling garden path effects without explicit hierarchical syntax. In 40th Annual Meeting of the Cognitive Science Society: Changing Minds, CogSci 2018, pages 2603–2608. The Cognitive Science Society. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2024-tree-transformer-s-disambiguation-ability/