Elamur Research⌕
☰
home / papers / 2019 / The ERG at MRP 2019: Radically Compositional Semantic Dependencies

The ERG at MRP 2019: Radically Compositional Semantic Dependencies

Paper · 2019 · Stephan Oepen and Dan Flickinger

Summary

This paper offers a reference point outside the official MRP 2019 competition. It asks how a hand-built, linguistically rich grammar compares with purely data-driven parsers on two target frameworks derived from English Resource Semantics: EDS and DM. The authors parsed the raw evaluation strings in one-best mode with the English Resource Grammar (ERG, release 1214) and the PET chart parser from the LOGON distribution. A discriminative parse-ranking model trained on the Redwoods treebank chose the best analysis. Each selected HPSG derivation fixes an underspecified logical form, which existing converters then turn into EDS and DM graphs. Because the conversion is formal, the resulting graphs follow the grammar's compositional algebra. Measured with MRP metrics, the ERG reached the highest overall F1 on both frameworks, ahead of the best shared-task systems. Its advantage was largest on node-local information such as labels, properties and anchors. Some data-driven systems matched or beat it on top nodes and edges. The authors read this as evidence that general-purpose grammatical knowledge helps when parsing into these meaning representations.

Key points

  • The ERG is an HPSG implementation begun in 1993. Its hand-built lexicon has about 38,000 lemmata covering 27,000 citation forms. With NER and unknown-word handling, it fully analyses 85–95% of utterances in standard corpora.
  • The Redwoods Ninth Growth treebank, matching ERG 1214, holds gold analyses for roughly 85,400 utterances, close to 1.3 million tokens, across about six genres. This includes WSJ sections from the PTB. The ERG and Redwoods together represent more than thirty person-years of work.
  • Parse ranking used a Maximum Entropy model with the Zhang et al. (2007) features, trained on Redwoods sections 00–20.
  • Overall F1 on the full evaluation data:
    • DM: ERG about .961, SJTU–NICT .955, HIT-SCIR .951.
    • EDS: ERG about .952, Peking .945.
    • DM node labels: ERG .987 vs. .949 for the best data-driven system.
  • Parsing speed was about two sentences per second on the short Little Prince sentences and about four seconds per sentence on longer WSJ text. The novel-style Little Prince subset exposed cross-domain drops for some data-driven parsers.

Interesting ideas

  • I1. Annotators never pick a whole tree among hundreds. Instead, they judge small isolated contrasts ("discriminants"), such as noun vs. verb reading or where a PP attaches, which makes manual disambiguation fast and reliable.
  • I2. A "dynamic treebank" stores the core analysis and projects several views from it (derivation, phrase tree, logical form, dependency graphs). When the grammar changes, it is re-updated mostly automatically, so earlier manual decisions are preserved.
  • I3. Because the meaning graph is derived deterministically from the syntactic derivation, every node and edge is guaranteed to come from explicit composition rules rather than learned guesses.
  • I4. The evaluation scores each graph component type separately (tops, labels, properties, anchors, edges). This exposes systems that excel locally but err structurally, and vice versa.
  • I5. A small out-of-domain subset, short literary sentences, was evaluated alongside the main newswire data to reveal domain sensitivity and the effect of input complexity.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver-data disagreements between witness parsers could be split into individual discriminants. An LLM teacher or a human would then resolve only the contested contrasts rather than reviewing entire trees.
  • I2 (tree): Annotation decisions could be stored independently of a given UD dialect or converter version. Treebanks could then be regenerated after conversion-rule changes without losing curated choices.
  • I3 (frames): Frame construction could be a strict compositional function of the UD tree. Each role assignment would then be traceable to a specific rule and subtree, fitting the explainability goal.
  • I4 (learn): The frame and world stages could report separate scores for predicates, roles, anchoring spans and attachment. This would show whether errors come from lexical choice or from structure.
  • I5 (answer): Fables and tales could be kept as a distinct held-out slice next to textbook material. This would detect when gains on one genre fail to transfer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States310 works · 284,480 citations🎓University of OsloOslo · Norway29 works · 3,700 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Stephan Oepen and Dan Flickinger. 2019. The ERG at MRP 2019: Radically Compositional Semantic Dependencies. Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k19-2003

@inproceedings{oepen2019erg,
  title = {{The ERG at MRP 2019: Radically Compositional Semantic Dependencies}},
  author = {Stephan Oepen and Dan Flickinger},
  booktitle = {Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning},
  year = {2019},
  doi = {10.18653/v1/k19-2003},
  url = {https://doi.org/10.18653/v1/k19-2003}
}

References

29 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hongxiao Bai and Hai Zhao. 2019. SJTU at MRP 2019: A transition-based multi-task parser for cross-framework meaning representation parsing. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 86 – 94, Hong Kong, China. find
  2. Emily M. Bender, Dan Flickinger, Stephan Oepen, Woodley Packard, and Ann Copestake. 2015. Layers of interpretation. On grammar and compositionality. In Proceedings of the 11th International Conference on Computational Semantics, pages 239 – 249, London, UK. find
  3. Ulrich Callmeier. 2002. Preprocessing and encoding techniques in PET. In Collaborative Language Engineering. A Case Study in Efficient Grammar-based Processing, pages 127 – 140, Stanford, CA. CSLI Publications. find
  4. John Carroll and Stephan Oepen. 2005. High-efficiency realization for a wide-coverage unification grammar. In Proceedings of the 2nd International Joint Conference on Natural Language Processing, pages 165 – 176, Jeju, Korea. find
  5. David Carter. 1997. The TreeBanker. A tool for supervised training of parsed corpora. In Proceedings of the Workshop on Computational Environments for Grammar Development and Linguistic Engineering, pages 9 – 15, Madrid, Spain. find
  6. Wanxiang Che, Longxu Dou, Yang Xu, Yuxuan Wang, Yijia Liu, and Ting Liu. 2019. HIT-SCIR at MRP 2019: A unified pipeline for meaning representation parsing via efficient training and effective encoding. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 76 – 85, Hong Kong, China. find
  7. Yufei Chen, Yajie Ye, and Weiwei Sun. 2019. Peking at MRP 2019: Factorization- and composition-based parsing for Elementary Dependency Structures. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 166 – 176, Hong Kong, China. Reflections As long-term co-developers of the ERG and its PET parser, we are impressed by the overall performance levels of the data-driven submissions to the MRP 2019 shared task. We hope to conduct more Ann Copestake, Dan Flickinger, Carl Pollard, and Ivan A. Sag. 2005. Minimal Recursion Semantics. An introduction. Research on Language and Computation, 3(4):281 – 332. 3 The Peking submission is not considered in the primary ranking of the official shared task results, because the team inadvertently used tokenization training data beyond the ‘whitelisted’ resources for task participants Gregor Erbach. 1991. A flexible parser for a linguistic development environment. In Otthein Herzog and Claus-Rainer Rollinger, editors, Text Understanding in LILOG, pages 74 – 87. Springer, Berlin, Germany. find
  8. Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English. The Penn Treebank. Computational Linguistics, 19:313 – 330.
  9. Dan Flickinger. 2000. On building a more efficient grammar by exploiting types. Natural Language Engineering, 6 (1):15 – 28.
  10. Stephan Oepen, Omri Abend, Jan Hajič, Daniel Hershcovich, Marco Kuhlmann, Tim O’Gorman, Nianwen Xue, Jayeol Chun, Milan Straka, and Zdeňka Urešová. 2019. MRP 2019: Cross-framework Meaning Representation Parsing. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 1 – 27, Hong Kong, China. find
  11. Dan Flickinger, Emily M. Bender, and Stephan Oepen. 2014. Towards an encyclopedia of compositional semantics. Documenting the interface of the English Resource Grammar. In Proceedings of the 9th International Conference on Language Resources and Evaluation, pages 875 – 881, Reykjavik, Iceland. find
  12. Dan Flickinger, Stephan Oepen, and Emily M. Bender. 2017. Sustainable development and refinement of complex linguistic annotations at scale. In Handbook of Linguistic Annotation, pages 353 – 377, Dordrecht, The Netherlands. Springer. ↗
  13. Stephan Oepen and Ulrich Callmeier. 2000. Measure for measure: Parser cross-fertilization. Towards increased component comparability and exchange. In Proceedings of the 6th International Conference on Parsing Technologies, pages 183 – 194, Trento, Italy. find
  14. Jean Mark Gawron, Jonathan King, John Lamping, Egon Loebner, E. Anne Paulson, Geoffrey K. Pullum, Ivan A. Sag, and Thomas Wasow. 1982. Processing English with a Generalized Phrase Structure Grammar. In Proceedings of the 20th Meeting of the Association for Computational Linguistics, pages 74 – 81, Toronto, Canada. find
  15. Stephan Oepen and John Carroll. 2000. Ambiguity packing in constraint-based parsing. Practical results. In Proceedings of the 1st Meeting of the North American Chapter of the Association for Computational Linguistics, pages 162 – 169, Seattle, WA, USA. find
  16. Angelina Ivanova, Stephan Oepen, Lilja Øvrelid, and Dan Flickinger. 2012. Who did what to whom? A contrastive study of syntacto-semantic dependencies. In Proceedings of the 6th Linguistic Annotation Workshop, pages 2 – 11, Jeju, Republic of Korea. find
  17. Stephan Oepen, Daniel Flickinger, Kristina Toutanova, and Christopher D. Manning. 2004. LinGO Redwoods. A rich and dynamic treebank for HPSG. Research on Language and Computation, 2(4):575 – 596.
  18. Stephan Oepen and Jan Tore Lønning. 2006. Discriminant-based MRS banking. In Proceedings of the 5th International Conference on Language Resources and Evaluation, pages 1250 – 1255, Genoa, Italy.
  19. Mark Johnson, Stuart Geman, Stephen Canon, Zhiyi Chi, and Stefan Riezler. 1999. Estimators for stochastic ‘unification-based’ grammars. In Proceedings of the 37th Meeting of the Association for Computational Linguistics, pages 535 – 541, College Park, MD, USA.
  20. Carl Pollard and Ivan A. Sag. 1994. Head-Driven Phrase Structure Grammar. Studies in Contemporary Linguistics. The University of Chicago Press, Chicago, USA. find
  21. Martin Kay. 1986. Algorithm schemata and data structures in syntactic processing. In In Barbara J. Grosz, Karen Spärck Jones, and Bonnie Lynn Weber, editors, Readings in Natural Language Processing, pages 35 – 70. Morgan Kaufmann, San Francisco, CA, USA. find
  22. Hideto Tomabechi. 1995. Design of efficient unification for natural language. Journal of Natural Language Processing, 2(2):23 – 58. ↗
  23. Kristina Toutanova, Christopher D. Manning, Dan Flickinger, and Stephan Oepen. 2005. Stochastic HPSG Parse Disambiguation using the Redwoods Corpus. Research on Language and Computation, 3:83 – 105. ↗
  24. Bernd Kiefer, Hans-Ulrich Krieger, John Carroll, and Robert Malouf. 1999. A bag of useful techniques for efficient and robust parsing. In Proceedings of the 37th Meeting of the Association for Computational Linguistics, pages 473 – 480, College Park, MD, USA. find
  25. Yi Zhang, Stephan Oepen, and John Carroll. 2007. Efficiency in unification-based n-best parsing. In Proceedings of the 10th International Conference on Parsing Technologies, pages 48 – 59, Prague, Czech Republic. find
  26. Jan Tore Lønning and Stephan Oepen. 2006. Re-usable tools for precision machine translation. In Proceedings of the COLING|ACL 2006 Interactive Presentation Sessions, pages 53 – 56, Sydney, Australia. find
  27. Yue Zhang, Wei Jiang, Qingrong Xia, Junjie Cao, Rui Wang, Zhenghua Li, and Min Zhang. 2019. Suda– alibaba at MRP 2019: Graph-based models with BERT. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 149 – 157, Hong Kong, China. find
  28. Robert Malouf. 2002. A comparison of algorithms for maximum entropy parameter estimation. In Proceedings of the 6th Conference on Natural Language Learning, pages 49 – 55, Taipei, Taiwan.
  29. Robert Malouf, John Carroll, and Ann Copestake. 2000. Efficient feature structure operations without compilation. Natural Language Engineering, 6(1):29 – 46. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oepen-2019-erg-mrp-2019-radically-compositional/