Elamur Research⌕
☰
home / papers / 2023 / SMATCH++: Standardized and Extended Evaluation of Semantic Graphs

SMATCH++: Standardized and Extended Evaluation of Semantic Graphs

Paper · 2023 · Juri Opitz

Summary

SMATCH is the standard way to compare semantic graphs such as AMR. It aligns the variables of two graphs and counts the triples they share. Its main uses are ranking parsers and checking generation output. The paper argues that this measurement is underspecified in ways that threaten fair comparison:

  • Preprocessing choices are hidden.
  • The usual hill-climbing aligner can stall in local optima and gives no bound on how far it is from the best alignment.
  • Extensions such as fine-grained aspect scores are scattered and inconsistently defined.

The author splits the metric into three modules (preprocessing, alignment, scoring) and treats each in turn:

  • Preprocessing: graphs are standardized so that structurally different but guideline-equivalent encodings (e.g., reified vs. non-reified relations) score the same.
  • Alignment: an ILP solver delivers optimal alignments in a standard evaluation setting, and a lossless graph compression shrinks the search space.
  • Scoring: aspect-level scores are computed over full extracted sub-graphs and extended to new phenomena.

The resulting SMATCH++ works for semantic graph evaluation much as SacreBLEU does for BLEU: it is a reproducible, extensible reference implementation.

Key points

  • A generalized triple-matching function places earlier variants (embedding-based concept matching, root-distance weighting) in one framework. Plain SMATCH is recovered with unit weights and exact label identity.
  • Reification and de-reification rules, which the AMR guidelines allow, are applied during preprocessing. Without this, equivalent graphs could be penalized for differing only in form.
  • ILP alignment found optimal solutions in reasonable time on the benchmark, including graphs with up to 174 variables. Lossless compression made the search notably more efficient.
  • Fine-grained scoring now takes whole sub-graphs, so two named cats with different names score 0.5 instead of a perfect match. New aspects were added: cause, location, quantification, and tense.
  • Six parsers were compared on these aspects (solver: ILP):
    • Cause is hardest, at 31.2–47.8 F1.
    • Tense ranges from 48.4 to 67.7.
    • Some parser rankings flip between aspects.
    • Gaps between parsers are wider than on full-graph scores.

Interesting ideas

  • I1. Use an exact solver that reports optimality, or at least an upper bound, instead of a heuristic search whose quality is unknown; otherwise a score difference may reflect the search rather than the systems.
  • I2. Normalize inputs into a canonical form before comparing, so that meaning-equivalent encodings permitted by the guidelines are not counted as errors.
  • I3. For each phenomenon, compare the entire sub-graph rooted at the triggering edge or node rather than a single label, which yields graded and more honest aspect scores.
  • I4. Report a breakdown by aspect (cause, tense, location, quantity) instead of one aggregate number, since rankings can flip by phenomenon and aggregates hide weak spots.
  • I5. Collect the metric's variants into a single parameterized function with weights and a similarity hook, so that extensions such as graded concept similarity stay comparable to the base metric.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When scoring Mova's frame graphs against gold or silver graphs, use exact alignment (ILP) so that regressions found in evaluation are real and not artifacts of the aligner.
  • I2 (tree): Converting UD dialects and treebank variants into one canonical form before computing LAS or comparing witness parsers would stop spurious disagreements from contaminating silver data.
  • I3 (world): Entity, time, and causal sub-structures in the world model could each be scored this way, showing exactly which kinds of knowledge the pipeline gets wrong.
  • I4 (answer): Breaking down QA and word-problem failures by the frame aspect involved would show whether errors come from causality, time, or quantities.
  • I5 (back): A text → graph → text round trip could be scored with this generalized match, using WordNet-based graded similarity on concepts, to measure how much meaning the compressor preserves.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Heidelberg UniversityHeidelberg · Germany38 works · 41,356 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Juri Opitz. 2023. SMATCH++: Standardized and Extended Evaluation of Semantic Graphs. Findings of the Association for Computational Linguistics: EACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-eacl.118

@inproceedings{opitz2023smatch,
  title = {{SMATCH++: Standardized and Extended Evaluation of Semantic Graphs}},
  author = {Juri Opitz},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-eacl.118},
  url = {https://doi.org/10.18653/v1/2023.findings-eacl.118}
}

References

43 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rafael Torres Anchiêta, Marco Antonio Sobrevilla Cabezudo, and Thiago Alexandre Salgueiro Pardo. 2019. Sema: an extended semantic evaluation for amr. In (To appear) Proceedings of the 20th Computational Linguistics and Intelligent Text Processing. Springer International Publishg. find
  2. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics.
  3. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2019. Amr guidlines. https://github.com/amrisi/amr-guidelines/blob/master/amr.md. ↗
  4. Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021. One spring to rule them both: Symmetric amr semantic parsing and generation without a complex pipeline. In Proceedings of AAAI. ↗
  5. Deng Cai and Wai Lam. 2019. Core semantic first: A top-down approach for AMR parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3799–3809, Hong Kong, China. Association for Computational Linguistics. ↗
  6. Deng Cai and Wai Lam. 2020. AMR parsing via graph-sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1290–1301, Online. Association for Computational Linguistics. ↗
  7. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics. find
  8. Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for Abstract Meaning Representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 536–546, Valencia, Spain. Association for Computational Linguistics. ↗
  9. Bradley Efron. 1992. Bootstrap methods: another look at the jackknife. Springer. ↗
  10. Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A. Smith. 2014. A discriminative graph-based parser for the Abstract Meaning Representation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426–1436, Baltimore, Maryland. Association for Computational Linguistics. ↗
  11. Michael Wayne Goodman. 2019. AMR normalization for fairer evaluation. CoRR, abs/1909.01568. ↗
  12. Thanh Lam Hoang, Gabriele Picco, Yufang Hou, Young-Suk Lee, Lam Nguyen, Dzung Phan, Vanessa López, and Ramon Fernandez Astudillo. 2021a. Ensembling graph predictions for amr parsing. Advances in Neural Information Processing Systems, 34. ↗
  13. Thanh Lam Hoang, Gabriele Picco, Yufang Hou, Young-Suk Lee, Lam Nguyen, Dzung Phan, Vanessa Lopez, and Ramon Fernandez Astudillo. 2021b. Ensembling graph predictions for amr parsing. In Advances in Neural Information Processing Systems, volume 34, pages 8495–8505. Curran Associates, Inc. ↗
  14. Alexander Miserlis Hoyle, Ana Marasović, and Noah A. Smith. 2021. Promoting graph awareness in linearized graph-to-text generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 944–956, Online. Association for Computational Linguistics. find
  15. Gunnar W Klau. 2009. A new graph-based method for pairwise global network alignment. BMC bioinformatics, 10(1):1–9. ↗
  16. Zi Lin, Jeremiah Liu, and Jingbo Shang. 2022. Neural-symbolic inference for robust autoregressive graph parsing via compositional uncertainty quantification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4759–4776, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  17. Matthias Lindemann, Jonas Groschwitz, and Alexander Koller. 2020. Fast semantic parsing with well-typedness guarantees. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3929–3951, Online. Association for Computational Linguistics. find
  18. Chunchuan Lyu. 2018. Fine-grained smatch implementation. https://github.com/ChunchuanLv/amr-evaluation-tool-enhanced. ↗
  19. Chunchuan Lyu and Ivan Titov. 2018. AMR parsing as graph prediction with latent alignment. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 397–407, Melbourne, Australia. Association for Computational Linguistics. ↗
  20. Emma Manning and Nathan Schneider. 2021. Referenceless parsing-based evaluation of AMR-to-English generation. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 114–122, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  21. Abelardo Carlos Martínez Lorenzo, Marco Maru, and Roberto Navigli. 2022. Fully-Semantic Parsing and Generation: the BabelNet Meaning Representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1727–1741, Dublin, Ireland. Association for Computational Linguistics. find
  22. Jonathan May and Jay Priyadarshi. 2017. Semeval-2017 task 9: Abstract meaning representation parsing and generation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 536–545. find
  23. Stephan Oepen, Omri Abend, Lasha Abzianidze, Johan Bos, Jan Hajic, Daniel Hershcovich, Bin Li, Tim O’Gorman, Nianwen Xue, and Daniel Zeman. 2020. Mrp 2020: The second shared task on cross-framework and cross-lingual meaning representation parsing. In Proceedings of the CoNLL 2020 Shared Task: Cross-Framework Meaning Representation Parsing, pages 1–22.
  24. Juri Opitz, Angel Daza, and Anette Frank. 2021. Weisfeiler-leman in the bamboo: Novel AMR graph metrics and a benchmark for AMR graph similarity. Transactions of the Association for Computational Linguistics, 9:1425–1441. ↗
  25. Juri Opitz and Anette Frank. 2021. Towards a decomposable metric for explainable evaluation of text generation from AMR. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1504–1518, Online. Association for Computational Linguistics. ↗
  26. Juri Opitz and Anette Frank. 2022a. Better Smatch = better parser? AMR evaluation is not so simple anymore. In Proceedings of the 3rd Workshop on Evaluation and Comparison of NLP Systems, pages 32–43, Online. Association for Computational Linguistics. find
  27. Juri Opitz and Anette Frank. 2022b. SBERT studies meaning representations: Decomposing sentence embeddings into explainable semantic features. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 625–638, Online only. Association for Computational Linguistics. find
  28. Juri Opitz, Letitia Parcalabescu, and Anette Frank. 2020. AMR Similarity Metrics from Principles. Transactions of the Association for Computational Linguistics, 8:522–538. ↗
  29. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71– 106. ↗
  30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  31. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics. ↗
  32. Leonardo F. R. Ribeiro, Yue Zhang, and Iryna Gurevych. 2021. Structural adapters in pretrained language models for AMR-to-Text generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4269–4282, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  33. Linfeng Song and Daniel Gildea. 2019. SemBleu: A robust metric for AMR parsing evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4547– 4552, Florence, Italy. Association for Computational Linguistics. ↗
  34. Elias Stengel-Eskin, Aaron Steven White, Sheng Zhang, and Benjamin Van Durme. 2020. Universal decompositional semantic parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8427–8439, Online. Association for Computational Linguistics. find
  35. Matteo Togninalli, Elisabetta Ghisu, Felipe Llinares-López, Bastian Rieck, and Karsten Borgwardt. 2019. Wasserstein weisfeiler-lehman graph kernels. In Advances in Neural Information Processing Systems, volume 32, pages 6436–6446. Curran Associates, Inc. ↗
  36. Sarah Uhrig, Yoalli Garcia, Juri Opitz, and Anette Frank. 2021. Translate, then parse! a strong baseline for cross-lingual AMR parsing. In Proceedings of the 17th International Conference on Parsing Technologies and the IWPT 2021 Shared Task on Parsing into Enhanced Universal Dependencies (IWPT 2021), pages 58–64, Online. Association for Computational Linguistics. find
  37. Rik van Noord, Lasha Abzianidze, Hessel Haagsma, and Johan Bos. 2018. Evaluating scoped meaning representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Languages Resources Association (ELRA). find
  38. Shira Wein, Wai Ching Leung, Yifu Mu, and Nathan Schneider. 2022. Effect of source language on AMR structure. In Proceedings of the 16th Linguistic Annotation Workshop (LAW-XVI) within LREC2022, pages 97–102, Marseille, France. European Language Resources Association. find
  39. Shira Wein and Nathan Schneider. 2022. Accounting for language effect in the evaluation of cross-lingual AMR parsers. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3824–3834, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. ↗
  40. Boris Weisfeiler and Andrei Leman. 1968. The reduction of a graph to canonical form and the algebra which appears therein. NTI, Series, 2(9):12–16. find
  41. Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, and Guodong Zhou. 2020. Improving AMR parsing with sequence-to-sequence pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2501– 2511, Online. Association for Computational Linguistics. find
  42. Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019. AMR parsing as sequence-to-graph transduction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 80–94, Florence, Italy. Association for Computational Linguistics. find
  43. Sheng Zhang, Xutai Ma, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2018. Cross-lingual decompositional semantic parsing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1664–1675, Brussels, Belgium. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/opitz-2023-smatch-standardized-extended-evaluation-semantic/