Elamur Research⌕
☰
home / papers / 2023 / Error Exploration for Automatic Abstract Meaning Representation Parsing

Error Exploration for Automatic Abstract Meaning Representation Parsing

Paper · 2023 · Maria Boritchev and Johannes Heinecke

Summary

The authors examine the failures of a multilingual AMR parser to see whether its errors can be diagnosed and fixed. The parser is AMRlib with T5 replaced by multilingual MT5. It was trained on English AMR 3.0 plus machine translations of that data into French, German, Spanish, Italian and Polish. For each language they parsed the AMR 3.0 development set, ranked the sentences by Smatch, and manually labelled the 50 worst parses. They used seven error categories, assigned in a fixed priority order so that each parse gets only one label. Coordination problems dominated in English and French, while translation problems dominated in the other languages. They then tried two fixes. The first reified relations in both gold and output graphs. The second split long sentences along UD dependency structure, parsed the pieces separately, and merged the resulting graphs. Some individual sentences improved a lot, but average Smatch barely changed. The paper is offered as a negative result, meant to serve as a reference point for future work on adapting and correcting parsers.

Key points

  • The parser is AMRlib (seq2seq) with MT5. It emits graphs without variables, and the variables are added in a postprocessing step. When the raw output is malformed, that step skips ahead to the first clean start, which leaves the graph incomplete.
  • Test Smatch: IT 73.9, ES 74.4, DE 71.0, FR 74.0, PL 72.2.
    • DE, ES and IT were evaluated on the manual translations in LDC2020T07.
    • FR and PL were evaluated on machine-translated test data.
  • The seven error categories, in priority order, are: translation, coordination, input, incomplete output, reification, gold error, other.
    • Coordination accounted for 42% of the worst English parses.
    • Translation accounted for 64% of the worst German parses.
  • The worst single-sentence Smatch per language ranged from 11.8 (PL) to 25.0 (EN).
  • Reifying every reifiable relation in both gold and system graphs raised English dev Smatch only from 85.4 to 86.0. Splitting sentences along the UD tree (parataxis adjuncts, relative clauses) left the average unchanged.

Interesting ideas

  • I1. Sort outputs by their per-sentence metric score, then hand-label only the bottom slice, using categories that are applied in a fixed order and are mutually exclusive, so the error counts are clean and comparable across languages.
  • I2. Before scoring, convert both the reference and the prediction into one canonical form (here, reified relations), to measure how much of the apparent error comes from the metric penalising equivalent notations.
  • I3. Break a long sentence at syntactic seams, such as parataxis or a relative clause, and replace the relative pronoun with its head noun so each piece becomes a full sentence. Parse the pieces separately, then join the graphs by unifying the shared head variable.
  • I4. Keep a separate "gold error" category, so that some bad scores are explicitly blamed on faulty annotation rather than on the system.
  • I5. Near-synonym concepts, such as smash-01 produced where gold has break-01, are scored as outright errors, which suggests evaluation should allow lexical closeness.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Run this routinely on the lowest-scoring silver or test sentences for each witness parser, so that repair-rule work goes to the categories that actually dominate.
  • I2 (learn): Normalise frame and graph variants on both sides before comparison, so that evaluation and silver-agreement checks do not count alternative but equivalent encodings as disagreement.
  • I3 (frames): Frame extraction could process relative clauses as independent clauses whose subject is the head noun, then link the frames through that shared entity, keeping each step explicit.
  • I4 (tree): When inspecting the lowest-LAS sentences in a treebank or UD dialect conversion, flag annotation mistakes in their own bucket and fix or exclude them instead of training the parser toward them.
  • I5 (world): Use WordNet/VerbNet class membership from the compiled ontology to give partial credit when predicted and expected predicates are close relatives.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢OrangeParis · France3 works · 21 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Machine translation · 1,087 worksEnglish · 2,263 worksSemantic parsing & meaning representation · 185 works

Cite

Cite as Maria Boritchev and Johannes Heinecke. 2023. Error Exploration for Automatic Abstract Meaning Representation Parsing. Proceedings of the 15th International Conference on Computational Semantics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{boritchev2023error,
  title = {{Error Exploration for Automatic Abstract Meaning Representation Parsing}},
  author = {Maria Boritchev and Johannes Heinecke},
  booktitle = {Proceedings of the 15th International Conference on Computational Semantics},
  year = {2023},
  url = {https://aclanthology.org/2023.iwcs-1.25/}
}

References

14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pages 178–186.
  2. Johan Bos. 2011. A survey of computational semantics: Representation, inference and knowledge in wide-coverage text understanding. Language and Linguistics Compass, 5(6):336–366. ↗
  3. Ida Szubert, Marco Damonte, Shay B Cohen, and Mark Steedman. 2020. The role of reentrancies in Abstract Meaning Representation parsing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2198–2207. find
  4. Shira Wein and Nathan Schneider. 2021. Classifying divergences in cross-lingual AMR pairs. In Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 56–65, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  5. Maja Buljan, Joakim Nivre, Stephan Oepen, and Lilja Øvrelid. 2022. A tale of four parsers: methodological reflections on diagnostic evaluation and in-depth error analysis for meaning representation parsing. Language Resources and Evaluation, 56(4):1075–1102. ↗
  6. Shira Wein and Nathan Schneider. 2022. Accounting for Language Effect in the Evaluation of Cross-lingual AMR Parsers. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3824–3834, Gyeongju, Republic of Korea. Association for Computational Linguistics. ↗
  7. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics. find
  8. Chen Yu and Daniel Gildea. 2022. Sequence-to-sequence AMR Parsing with Ancestor Information. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 571–577, Dublin, Ireland. Association for Computational Linguistics. ↗
  9. Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for abstract meaning representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 536–546, Valencia, Spain. Association for Computational Linguistics. ↗
  10. Zdeněk Žabokrtský, Daniel Zeman, and Magda Ševčíková. 2020. Sentence meaning representations across languages: what can we learn from existing frameworks? Computational Linguistics, 46(3):605– 665. ↗
  11. Zhenyun Deng, Yonghua Zhu, Yang Chen, Michael Witbrock, and Patricia Riddle. 2022. Interpretable amr-based question decomposition for multi-hop question answering. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22), pages 4093–4099, Vienna, Austria. find
  12. Kuan-Hao Huang, Varun Iyer, Anoop Kumar, Sriram Venkatapathy, Kai-Wei Chang, and Aram Galstyan. 2022. Unsupervised syntactically controlled paraphrase generation with Abstract Meaning Representations. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1547–1554, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  13. Paul Kingsbury and Martha Palmer. 2002. From Tree-Bank to PropBank. In Proceedings of the Third International Conference on Language Resources and Evaluation, Las Palmas, Canary Islands - Spain. European Language Resources Association. find
  14. Changmao Li and Jeffrey Flanigan. 2022. Improving neural machine translation with the Abstract Meaning Representation by combining graph and sequence transformers. In Proceedings of the 2nd Workshop on Deep Learning on Graphs for Natural Language Processing (DLG4NLP 2022), pages 12–21, Seattle, Washington. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/boritchev-2023-error-exploration-automatic-abstract-meaning/