Elamur Research⌕
☰
home / papers / 2024 / The Relative Clauses AMR Parsers Hate Most

The Relative Clauses AMR Parsers Hate Most

Paper · 2024 · Xiulin Yang and Nathan Schneider

Summary

Headline Smatch scores for English AMR parsers are high, but they hide how parsers handle specific hard constructions. This paper looks at one frequent long-distance dependency: relative clauses (RCs). In these, the modified noun also fills an argument slot inside the clause.

The authors evaluate five AMR parsers on three sets: naturalistic sentences from the UD English Web Treebank (EWT), the synthetic Controlled RCs (CRC) set, and the AMR 3.0 test split. They compare structure-unaware seq2seq parsers with a structure-aware compositional one. Inverse "-of" edges are flipped so that RCs show up as reentrancies, and recall of those reentrancies is scored against Enhanced UD annotations. RCs are sorted into six subtypes, and EUD edges for reduced RCs are added automatically.

All parsers struggle: on EWT, reentrancy recall stays below 64%. The seq2seq models beat the compositional model, whether or not they were trained with structural biases. Passive subject RCs are the easiest, while oblique and reduced RCs cause the most errors. The study shows that aggregate metrics can mask systematic gaps in predicate-argument recovery.

Key points

  • Evaluation sets:
    • EWT (train split): 1,449 RC sentences, mean length 26.5 tokens, with no overlap with AMR 3.0 confirmed.
    • CRC: 1,400 sentences, mean length 13.7, covering 4 RC types with 350 each.
    • AMR 3.0 test: 259 sentences, mean length 29.1.
  • The six RC subtypes are subject, object, oblique, passive, reduced object and reduced oblique. Free relatives, possessive RCs, reduced subject RCs and advcl:relcl clefts are left out because they are rare.
  • In EUD, the head noun receives the clause-internal function, and the relative pronoun links to the head via a ref edge. In UD, the RC predicate attaches with acl:relcl.
  • Reentrancy recall on EWT is under 64% for every parser. Seq2seq models outperform the compositional parser.
  • Difficulty ranking: passive subject RCs are easiest, while oblique and reduced RCs are hardest.

Interesting ideas

  • I1. Flipping inverse role edges (ARG0-of → ARG0) turns hidden argument sharing into explicit reentrancies, so long-distance arguments can be scored directly rather than buried in a global score.
  • I2. Enhanced UD gives a ready-made gold signal for the noun's role inside the relative clause, and the authors infer such edges automatically even when no relativizer is present.
  • I3. A naturalistic treebank is paired with a controlled synthetic set, balanced across constructions, to separate the effect of construction type from confounds like sentence length.
  • I4. Results are broken down by construction subtype, which exposes that oblique and reduced RCs fail far more often than passives, a pattern a single aggregate number would hide.
  • I5. A treebank's training split is reused as a test set for systems never trained on it, after an explicit overlap check, which yields far more examples than a dev or test split.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): normalise inverse roles before checking whether the frame of an RC verb received the head noun as its argument, and report that recall separately.
  • I2 (tree): add a rule that produces enhanced edges for reduced object and oblique relatives, so that downstream frames do not lose the argument when "that/which" is missing.
  • I3 (learn): build small templated RC suites alongside real-text evaluation, so that a drop can be traced to the construction itself rather than to long sentences.
  • I4 (answer): tag test questions on fables by the RC type they depend on, to see whether wrong answers cluster on "the fox the crow gave cheese to"-style reduced obliques.
  • I5 (learn): when evaluating witness parsers or LLM teachers on constructions, use large splits they never saw, verified by overlap checks, to get enough rare-subtype instances.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksNeuroscience · 165 worksCognitive Neuroscience · 164 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSemantic parsing & meaning representation · 185 worksUniversal Dependencies · 184 works

Cite

Cite as Xiulin Yang and Nathan Schneider. 2024. The Relative Clauses AMR Parsers Hate Most. Proceedings of the Fifth International Workshop on Designing Meaning Representations @ LREC-COLING 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yang2024relative,
  title = {{The Relative Clauses AMR Parsers Hate Most}},
  author = {Xiulin Yang and Nathan Schneider},
  booktitle = {Proceedings of the Fifth International Workshop on Designing Meaning Representations @ LREC-COLING 2024},
  year = {2024},
  doi = {10.63317/2ystgqwwgrte},
  url = {https://doi.org/10.63317/2ystgqwwgrte}
}

References

37 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022. Graph pre-training for AMR parsing and generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6001–6015, Dublin, Ireland. Association for Computational Linguistics.
  2. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics.
  3. Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English Web Treebank. LDC2012T13. find
  4. Austin Blodgett and Nathan Schneider. 2021. Probabilistic, structure-aware algorithms for improved variety, accuracy, and coverage of AMR alignments. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3310–3321, Online. Association for Computational Linguistics. ↗
  5. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics. find
  6. Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1914–1925, Brussels, Belgium. Association for Computational Linguistics. ↗
  7. Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for Abstract Meaning Representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 536–546, Valencia, Spain. Association for Computational Linguistics. ↗
  8. Forrest Davis and Marten van Schijndel. 2020. Recurrent neural network language models always learn English-like relative clause attachment. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1979–1990, Online. Association for Computational Linguistics. ↗
  9. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308.
  10. Holger Diessel and Michael Tomasello. 2005. A new look at the acquisition of relative clauses. Language, pages 882–906. ↗
  11. Jamie Y. Findlay and Dag T. T. Haug. 2021. How useful are enhanced Universal Dependencies for semantic interpretation? In Proceedings of the Sixth International Conference on Dependency Linguistics (Depling, SyntaxFest 2021), pages 22–34, Sofia, Bulgaria. Association for Computational Linguistics.
  12. Edward Gibson. 1998. Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1):1– 76. ↗
  13. Peter C Gordon and Matthew W Lowder. 2012. Complex sentence processing: A review of theoretical perspectives on the comprehension of relative clauses. Language and Linguistics Compass, 6(7):403–415. ↗
  14. Jonas Groschwitz, Shay Cohen, Lucia Donatelli, and Meaghan Fowlie. 2023. AMR parsing is far from solved: GrAPES, the granular AMR parsing evaluation suite. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10728–10752, Singapore. Association for Computational Linguistics. ↗
  15. Jonas Groschwitz, Matthias Lindemann, Meaghan Fowlie, Mark Johnson, and Alexander Koller. 2018. AMR dependency parsing with a typed semantic algebra. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1831–1841, Melbourne, Australia. Association for Computational Linguistics. ↗
  16. Franny Hsiao and Edward Gibson. 2003. Processing relative clauses in Chinese. Cognition, 90(1):3–27. ↗
  17. Edward L. Keenan and Bernard Comrie. 1977. Noun phrase accessibility and universal grammar. Linguistic Inquiry, 8(1):63–99. find
  18. Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 235–249, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  19. Paul Kingsbury and Martha Palmer. 2002. From TreeBank to PropBank. In Proc. of LREC, pages 1989–1993, Las Palmas, Canary Islands.
  20. Kevin Knight, Bianca Badarau, Laura Baranescu, Claire Bonial, Madalina Bardocz, Kira Griffitt, Ulf Hermjakob, Daniel Marcu, Martha Palmer, Tim O’Gorman, and Nathan Schneider. 2021. Abstract Meaning Representation (AMR) Annotation Release 3.0. Linguistic Data Consortium, LDC2020T02. find
  21. Young-Suk Lee, Ramón Astudillo, Hoang Thanh Lam, Tahira Naseem, Radu Florian, and Salim Roukos. 2022. Maximum Bayes Smatch ensemble distillation for AMR parsing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5379–5392, Seattle, United States. Association for Computational Linguistics. ↗
  22. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. ↗
  23. Bingzhi Li, Lucia Donatelli, Alexander Koller, Tal Linzen, Yuekun Yao, and Najoung Kim. 2023. SLOG: A structural generalization benchmark for semantic parsing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3213–3232, Singapore. Association for Computational Linguistics. find
  24. Haitao Liu, Chunshan Xu, and Junying Liang. 2017. Dependency distance: A new perspective on syntactic patterns in natural languages. Physics of life reviews, 21:171–193. ↗
  25. Marius Mosbach, Stefania Degaetano-Ortlieb, Marie-Pauline Krielke, Badr M. Abdullah, and Dietrich Klakow. 2020. A closer look at linguistic knowledge in masked language models: The case of relative clauses in American English. In Proceedings of the 28th International Conference on Computational Linguistics, pages 771–787, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  26. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proc. of LREC, pages 4027– 4036, Marseille, France. ↗
  27. Sameer Pradhan, Julia Bonn, Skatje Myers, Kathryn Conger, Tim O’Gorman, James Gung, Kristin Wright-Bettner, and Martha Palmer. 2022. PropBank comes of age—larger, smarter, and more diverse. In Proc. of *SEM, pages 278–288, Seattle, Washington.
  28. Grusha Prasad, Marten van Schijndel, and Tal Linzen. 2019. Using priming to uncover the organization of syntactic representations in neural language models. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 66–76, Hong Kong, China. Association for Computational Linguistics. ↗
  29. Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. 2021. Counterfactual interventions reveal the causal effect of relative clause representations on agreement prediction. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 194– 209, Online. Association for Computational Linguistics. find
  30. Florencia Reali and Morten H Christiansen. 2007. Processing of relative clauses is made easier by frequency of occurrence. Journal of memory and language, 57(1):1–23. ↗
  31. Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 922–938, Online. Association for Computational Linguistics. find
  32. Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel R. Bowman, Miriam Connor, John Bauer, and Christopher D. Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), pages 2897–2904, Reykjavík, Iceland. find
  33. Nicolas Spring, Annette Rios, and Sarah Ebling. 2021. Exploring German multi-level text simplification. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1339– 1349, Held Online. INCOMA Ltd. find
  34. Ida Szubert, Marco Damonte, Shay B. Cohen, and Mark Steedman. 2020. The role of reentrancies in Abstract Meaning Representation parsing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2198–2207, Online. Association for Computational Linguistics. find
  35. David Temperley. 2007. Minimization of dependency length in written English. Cognition, 105(2):300–333. ↗
  36. Alex Warstadt and Samuel R Bowman. 2019. Linguistic analysis of pretrained sentence encoders with acceptability judgments. arXiv preprint arXiv:1901.03438. ↗
  37. Yuekun Yao and Alexander Koller. 2022. Structural generalization is hard for sequence-to-sequence models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5048–5062, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2024-relative-clauses-amr-parsers-hate/