Elamur Research⌕
☰
home / papers / 2023 / AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite

AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite

Paper · 2023 · Jonas Groschwitz, Shay B. Cohen, Lucia Donatelli et al.

Summary

Modern AMR parsers now reach Smatch scores close to or above reported inter-annotator agreement. Earlier human evaluation, however, judged only 62% of parser outputs acceptable, so a single overall score hides serious meaning errors. The authors respond with GrAPES, a challenge suite for English AMR parsing. It has 36 categories grouped into 9 sets, covering:

  • lexical sparsity (rare, unseen and frequent labels and senses),
  • several kinds of reentrancy, including pragmatic coreference,
  • structural generalization (deep recursion, long lists),
  • other phenomena such as PP attachment, ellipsis and long-range dependencies.

The data comes from four sources: hand-filtered slices of the AMR 3.0 test set, existing corpora, grammar-generated sentences and hand-written items. Each category gets its own metric, which looks only at the relevant part of the graph. Prerequisite checks and sanity checks separate failures on the target phenomenon from unrelated mistakes.

Results for three recent parsers show clear weaknesses in data sparsity, contextual disambiguation and structural generalization. The suite gives the community a finer, more trustworthy picture of what parsers can do and which components need work. It is released open source with visualization and table-generation code.

Key points

  • GrAPES has 36 categories in 9 sets. It reuses the Winograd Schema Challenge, "Putting words into BERT's mouth" (Karidi et al.) and the Unbounded Dependencies Corpus (Rimell et al.). Its grammar-generated sentences are built with Alto.
  • Category definitions are tied to frequency in the AMR 3.0 training data:
    • A predicate sense counts as "frequent" if it appears at least 30 times, and the other senses of the same lemma must also total at least 30.
    • A node label counts as "rare" if it appears 1–5 times.
    • CP recursion is tested up to depth 10, deeper than anything in the training data.
  • Prerequisite rates change how edge-recall numbers should be read. For AMRBart, PP attachment and Unaccusatives have similar edge recall (66 vs 71), but prerequisites are met 95% vs 79% of the time. Corrected for this, AMRBart handles Unaccusatives much better than PP attachment.
  • Sanity checks have limits. On Nested Control and Coordination, the AM Parser's 60% error rate came mostly from a lexical slip (linking "me" to an i node), not from structural failure.
  • GrAPES has no dev/test split, and part of it comes from the AMR test set. The authors therefore advise developing only on the non-test-derived parts and reporting any such use prominently.

Interesting ideas

  • I1. Each structural-generalization category has a shallow "sanity check" twin built from the same vocabulary, so a failure at depth can be blamed on structure rather than on unfamiliar words.
  • I2. Scores are conditioned on prerequisites: a phenomenon is graded only where the parser got the necessary anchor nodes right, so upstream errors are not counted twice.
  • I3. Sense-disambiguation items are kept only when the competing senses of the same lemma are also well attested, so the test genuinely requires a choice between readings.
  • I4. Pragmatic coreference is graded separately from coreference that grammar alone determines (gap reentrancies, repeated "I"), which shows how much real inference the system does.
  • I5. Each metric looks only at the annotated relevant subgraph instead of the whole output, which reduces noise from unrelated mistakes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Generate shallow and deep variants of the same templates (center embedding, CP recursion, long coordination) for the perceptron parser and gparse, and report the difference between them as the real generalization gap.
  • I2 (learn): Gate each evaluation item on whether its preconditions held (correct tokens, head and frame), and report the precondition rate next to the score so stages can be compared fairly.
  • I3 (frames): When testing VerbNet role or frame assignment, keep only lemmas whose alternative classes are common in our data, so a high score cannot come from always picking the majority class.
  • I4 (frames): Split coreference evaluation into syntactically forced and pragmatically resolved links, using Winograd-style pairs for the second group, so explainable resolution rules are credited only where they actually reason.
  • I5 (world): Score world-of-text tests by checking only the entity, state or event-order facts each item targets, so a stray error elsewhere does not hide whether the targeted inference worked.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MichiganAnn Arbor · United States86 works · 15,493 citations🔬RIKEN Center for Advanced Intelligence ProjectTokyo · Japan16 works · 957 citations🔬RIKENWako · Japan30 works · 607 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,379 worksSemantic parsing & meaning representation · 193 works

Cite

Cite as Jonas Groschwitz, Shay B. Cohen, Lucia Donatelli and Meaghan Fowlie. 2023. AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.662

@inproceedings{groschwitz2023amr,
  title = {{AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite}},
  author = {Jonas Groschwitz, Shay B. Cohen, Lucia Donatelli and Meaghan Fowlie},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.662},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.662}
}

References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022. Graph pre-training for AMR parsing and generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6001–6015, Dublin, Ireland. Association for Computational Linguistics.
  2. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics.
  3. Austin Blodgett and Nathan Schneider. 2021. Probabilistic, structure-aware algorithms for improved variety, accuracy, and coverage of AMR alignments. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3310–3321, Online. Association for Computational Linguistics. ↗
  4. Deng Cai and Wai Lam. 2019. Core semantic first: A top-down approach for AMR parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3799–3809, Hong Kong, China. Association for Computational Linguistics. ↗
  5. Deng Cai and Wai Lam. 2020. AMR parsing via graph-sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1290–1301, Online. Association for Computational Linguistics. ↗
  6. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics. find
  7. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. ↗
  8. Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N Mendes. 2013. Improving efficiency and accuracy in multilingual entity extraction. In Proceedings of the 9th international conference on semantic systems, pages 121–124. find
  9. Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for Abstract Meaning Representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 536–546, Valencia, Spain. Association for Computational Linguistics. ↗
  10. Johannes Gontrum, Jonas Groschwitz, Alexander Koller, and Christoph Teichmann. 2017. Alto: Rapid prototyping for parsing and translation. In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 29–32, Valencia, Spain. Association for Computational Linguistics. ↗
  11. Jonas Groschwitz. 2019. Methods for taking semantic graphs apart and putting them back together again. ↗
  12. Jonas Groschwitz, Matthias Lindemann, Meaghan Fowlie, Mark Johnson, and Alexander Koller. 2018. AMR dependency parsing with a typed semantic algebra. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1831–1841, Melbourne, Australia. Association for Computational Linguistics. ↗
  13. Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: the 90% solution. In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers, pages 57–60. find
  14. Taelin Karidi, Yichu Zhou, Nathan Schneider, Omri Abend, and Vivek Srikumar. 2021. Putting words in BERT’s mouth: Navigating contextualized vector spaces with pseudowords. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10300–10313, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  15. Najoung Kim and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087–9105, Online. Association for Computational Linguistics. find
  16. Kevin Knight, Bianca Badarau, Laura Baranescu, Claire Bonial, Madalina Bardocz, Kira Griffitt, Ulf Hermjakob, Daniel Marcu, Martha Palmer, Tim O’Gorman, et al. 2021. Abstract meaning representation (amr) annotation release 3.0. find
  17. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. find
  18. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. ↗
  19. Bingzhi Li, Lucia Donatelli, Alexander Koller, Tal Linzen, Yuekun Yao, and Najoung Kim. 2023. SLOG: A structural generalization benchmark for semantic parsing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Singapore. Association for Computational Linguistics. find
  20. Matthias Lindemann, Jonas Groschwitz, and Alexander Koller. 2020. Fast semantic parsing with well-typedness guarantees. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3929–3951, Online. Association for Computational Linguistics. find
  21. Juri Opitz. 2023. Smatch++: Standardized and extended evaluation of semantic graphs. arXiv preprint arXiv:2305.06993. ↗
  22. Ida Szubert, Marco Damonte, Shay B Cohen, and Mark Steedman. 2020. The role of reentrancies in Abstract Meaning Representation parsing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2198–2207. find
  23. Pia Weißenhorn, Lucia Donatelli, and Alexander Koller. 2022a. Compositional generalization with a broad-coverage semantic parser. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 44–54, Seattle, Washington. Association for Computational Linguistics. find
  24. Pia Weißenhorn, Yuekun Yao, Lucia Donatelli, and Alexander Koller. 2022b. Compositional generalization requires compositional parsers. arXiv preprint arXiv:2202.11937. ↗
  25. Edwin B Wilson. 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212. ↗
  26. Terry Winograd. 1971. Procedures as a representation for data in a computer program for understanding natural language. Technical report, MAS-SACHUSETTS INST OF TECH CAMBRIDGE PROJECT MAC. find
  27. Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero-shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6397–6407, Online. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/groschwitz-2023-amr-parsing-is-far-solved/