Elamur Research⌕
☰
home / papers / 2026 / TrAinMR: an Annotator Training Website for Abstract Meaning Representation

TrAinMR: an Annotator Training Website for Abstract Meaning Representation

Summary

Gold-standard Abstract Meaning Representation (AMR) corpora depend on human annotators. Even linguists need onboarding first, because the AMR guidelines are long and the role inventory is large. GPT-style automation has also proven too error-prone to replace manual annotation. The authors present TrAinMR, a web-based tool that standardizes this onboarding for any AMR or AMR-extension project. It has two parts. The first is a written Tutorial on PENMAN notation, PropBank framesets, roles and relations, reification, negation, modals and questions, illustrated with example graphs. The second is an Annotation Practice module where learners annotate 25 hand-picked AMR3.0 sentences of varying difficulty. After each submission, users can view the gold graph, hand-written explanations, a line-by-line breakdown and a highlighted comparison with their own answer. Malformed input triggers syntax error messages. In two pilot studies with five novice annotators each, most participants said their understanding of AMR improved, and some showed rising SMATCH scores during practice. The tool lowers the entry barrier for non-linguists and crowdworkers and may encourage new AMR corpora.

Key points

  • TrAinMR (Training in Abstract Meaning Representation) is described as the first tool built specifically to train AMR annotators. Existing tools such as CAMRA, LiDARR, Anafora and X-AMR are annotation interfaces, not training systems. The site is hosted at acnlplab.github.io/amr-training-site.
  • The Practice module uses 25 manually selected AMR3.0 sentences. They can be shown in random order (the default) or in order of increasing complexity, and the gold AMR stays hidden until the user submits.
  • Submissions are logged via the Google Sheets API, checked for valid PENMAN syntax and compared with gold graphs using SMATCH. Common errors flagged include mismatched parentheses, missing ":" on roles, and duplicate or omitted variables.
  • The Tutorial's topics follow Banarescu et al. (2013) and the NAACL 2015 AMR Tutorial slides.
  • The evaluation consisted of two pilots with 5 novices each (10 total), combining SMATCH scores, qualitative error analysis and post-study feedback. For context, released AMR datasets report inter-annotator agreement of 0.71–0.89 SMATCH.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2nzfbbesoii2

@inproceedings{yang2026trainmr,
  title = {{TrAinMR: an Annotator Training Website for Abstract Meaning Representation}},
  author = {Yang, Mina and Wein, Shira},
  booktitle = {Proceedings of The Seventh International Workshop on Designing Meaning Representations (DMR 2026) @ LREC 2026},
  year = {2026},
  doi = {10.63317/2nzfbbesoii2},
  url = {https://doi.org/10.63317/2nzfbbesoii2}
}

References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shafiuddin Rehan Ahmed, Jon Cai, Martha Palmer, and James H. Martin. 2024. X-AMR annotation tool. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 177–186, St. Julians, Malta. Association for Computational Linguistics. ↗
  2. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics. find our card
  3. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2019. Abstract meaning representation (amr) 1.2.6 specification. find
  4. Julia Bonn, Martha Palmer, Zheng Cai, and Kristin Wright-Bettner. 2020. Spatial AMR: Expanded spatial annotation in the context of a grounded Minecraft corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4883–4892, Marseille, France. European Language Resources Association. find
  5. Richard Brutti, Lucia Donatelli, Kenneth Lai, and James Pustejovsky. 2022. Abstract Meaning Representation for gesture. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1576–1583, Marseille, France. European Language Resources Association. find
  6. Jon Cai, Shafiuddin Rehan Ahmed, Julia Bonn, Kristin Wright-Bettner, Martha Palmer, and James H. Martin. 2023. CAMRA: Copilot for AMR annotation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 381–388, Singapore. Association for Computational Linguistics. ↗
  7. Jon Cai, Kristin Wright-Bettner, Zekun Zhao, Shafiuddin Rehan Ahmed, Abijith Trichur Ramachandran, Jeffrey Flanigan, Martha Palmer, and James Martin. 2025. LiDARR: Linking document AMRs with referents resolvers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 426–435, Vienna, Austria. Association for Computational Linguistics. find
  8. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics. find
  9. Wei-Te Chen and Will Styler. 2013. Anafora: A web-based general purpose annotation tool. In Proceedings of the 2013 NAACL HLT Demonstration Session, pages 14–19, Atlanta, Georgia. Association for Computational Linguistics. find
  10. Richard Durstenfeld. 1964. Algorithm 235: random permutation. Communications of the ACM, 7(7):420. ↗
  11. Allyson Ettinger, Jena Hwang, Valentina Pyatkin, Chandra Bhagavatula, and Yejin Choi. 2023. “you are an expert linguistic annotator”: Limits of LLMs as analyzers of Abstract Meaning Representation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8250–8263, Singapore. Association for Computational Linguistics. find
  12. Ulf Hermjakob. 2013. Amr editor: A tool to build abstract meaning representations. Marina del Rey, CA. USC Information Sciences Institute. find
  13. Kevin Knight, Bianca Badarau, Laura Banarescu, Claire Bonial, Madalina Bardocz, Kira Griffitt, Ulf Hermjakob, Daniel Marcu, Martha Palmer, Tim O’Gorman, et al. 2020. Abstract Meaning Representation (AMR) Annotation Release 3.0. Technical Report LDC2020T02, Linguistic Data Consortium, Philadelphia, PA. find
  14. Kexin Liao, Logan Lebanoff, and Fei Liu. 2018. Abstract Meaning Representation for multi-document summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1178–1190, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
  15. Mary Martin, Cecilia Mauceri, Martha Palmer, and Christoffer Heckman. 2020. Leveraging non-specialists for accurate and time efficient amr annotation. In Proceedings of the LREC 2020 Workshop on" Citizen Linguistics in Language Resource Development", pages 35–39. find
  16. Abelardo Carlos Martínez Lorenzo, Marco Maru, and Roberto Navigli. 2022. Fully-Semantic Parsing and Generation: the BabelNet Meaning Representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1727–1741, Dublin, Ireland. Association for Computational Linguistics. find
  17. Christian Matthiessen and John A Bateman. 1991. Text generation and systemic-functional linguistics: experiences from English and Japanese. Pinter Publishers. find
  18. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. ↗ our card
  19. Nathan Schneider, Tim O’Gorman, and Jeffrey Flanigan. 2015. The logic of amr practical, unified, graph-based sentence semantics for nlp. find
  20. Linfeng Song, Daniel Gildea, Yue Zhang, Zhiguo Wang, and Jinsong Su. 2019. Semantic neural machine translation using amr. Transactions of the Association for Computational Linguistics, 7:19–31. ↗ our card
  21. Jens EL Van Gysel, Meagan Vigus, Jayeol Chun, Kenneth Lai, Sarah Moeller, Jiarui Yao, Tim O’Gorman, Andrew Cowell, William Croft, Chu-Ren Huang, et al. 2021. Designing a uniform meaning representation for natural language processing. KI-Künstliche Intelligenz, 35(3):343– 360. ↗
  22. Shira Wein. 2025. Ambiguity and disagreement in Abstract Meaning Representation. In Proceedings of Context and Meaning: Navigating Disagreements in NLP Annotation, pages 145– 154, Abu Dhabi, UAE. International Committee on Computational Linguistics. find
  23. Shira Wein and Nathan Schneider. 2024. Assessing the cross-linguistic utility of abstract meaning representation. Computational Linguistics, 50(2):419–473. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-trainmr-annotator-training-website-abstract/