Elamur Research⌕
☰
home / papers / 2023 / Unified Syntactic Annotation of English in the CGEL Framework

Unified Syntactic Annotation of English in the CGEL Framework

Paper · 2023 · Brett Reynolds, Aryaman Arora and Nathan Schneider

Summary

The authors investigate if CGEL (Huddleston and Pullum, 2002), the well-known descriptive grammar of English, is usable in practice as a scheme for annotating corpora. CGEL is a 1700+ page descriptive reference grammar, but it was never designed or tested for treebanking, and it contains only 40 trees. The team turned its descriptions into an annotation manual of about 75 pages. In doing so, they settled points the grammar leaves open, such as when gaps may be used, how fused functions are represented, and how currency expressions are ordered. Each node in a CGEL tree carries both a category and a grammatical function, so one tree holds the information that PTB and UD each capture only partly. The authors released a small treebank, CGELBank, with validation and scoring code. They then ran an interannotator study on 50 sentences from the English Web Treebank, scored with tree-edit-distance metrics. Agreement was high, including on gaps. This suggests that CGEL-style trees are feasible, and they could later be derived automatically from existing treebanks as a more readable, linguistically documented view.

Key points

  • CGELBank, its validator and its IAA scripts are released at github.com/nert-nlp/cgel, along with an annotation manual of roughly 75 pages.
  • The POS tagset has 11 tags: N, N pro, V, V aux, P, D, Adj, Adv, Sdr, Coordinator and Int. Phrasal categories include Nom, NP, VP, Clause, Clause rel, PP, DP, AdjP, AdvP, IntP and Coordination.
  • Agreement was measured on 50 EWT trees:
    • Strict agreement was 91.6% and structure-only (unlab) agreement was 94.8%.
    • Flex agreement between the two annotators rose from 93.2% to 93.9% after validator feedback.
    • Gap agreement reached an F1 of 87.2% over 21 gaps.
    • Only 18% of trees were fully identical between annotators. Compared with the adjudicated set, 54% (annotator 1) and 32% (annotator 2) were identical.
  • Function disagreements outnumbered category disagreements (75 vs. 44). Among them were 18 Comp/Mod confusions and 11 Mod/Supplement confusions.
  • Annotators corrected trees proposed by an active-learning parser rather than building them from scratch.

Interesting ideas

  • I1. Each constituent gets two labels: a category for what kind of unit it is, and a function for its role in the parent. This merges constituency and dependency information in one structure.
  • I2. In fused constructions, one node plays two roles, e.g. which acting as both Determiner and Head. The longer of its two incoming edges is dropped because it can be rebuilt deterministically from the "Determiner-Head" label, so the structure stays a tree.
  • I3. A rule-based validator flags structural implausibilities such as missing or extra Nom levels and categories placed in unlikely functions, and it measurably improved agreement.
  • I4. Agreement is scored with tree edit distance, with fractional costs per error type and strict/flex/unlabeled variants, so disagreements are broken down by cause rather than collapsed into one number.
  • I5. Currency and similar expressions are annotated in the order they are spoken, not the order they are written, so over $300 is treated as "[over 300] dollars".

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Mova's UD trees could carry an extra CGEL-style function layer (e.g., Comp vs. Mod vs. Supplement) next to UD relations, giving the frames stage a cleaner argument/adjunct signal.
  • I2 (tree): Mova could keep its parser output strictly tree-shaped while encoding fused heads (the rich, those three) in labels, then expand them into graph edges when frames are built.
  • I3 (learn): a similar well-formedness checker could filter or flag silver trees from the witness parsers before training, catching spurious errors cheaply.
  • I4 (learn): the evaluation harness could report parser errors split into structure, category and function costs, showing which repair rules actually help.
  • I5 (tokens): the tokenizer could normalize currency symbols into pronunciation order, so measures and values attach correctly in later stages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgetown UniversityWashington · United States90 works · 4,732 citations🎓Humber PolytechnicToronto · Canada2 works · 2 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Brett Reynolds, Aryaman Arora and Nathan Schneider. 2023. Unified Syntactic Annotation of English in the CGEL Framework. LAW-XVII 2023 (Linguistic Annotation Workshop, ACL).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.law-1.22

@inproceedings{reynolds2023unified,
  title = {{Unified Syntactic Annotation of English in the CGEL Framework}},
  author = {Brett Reynolds, Aryaman Arora and Nathan Schneider},
  booktitle = {LAW-XVII 2023 (Linguistic Annotation Workshop, ACL)},
  year = {2023},
  doi = {10.18653/v1/2023.law-1.22},
  url = {https://doi.org/10.18653/v1/2023.law-1.22}
}

References

47 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven P. Abney. 1987. The English noun phrase in its sentential aspect. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA. find
  2. ES Atwell. 2008. Development of tag sets for part-of-speech tagging. In Corpus Linguistics: An International Handbook, volume 1, pages 501–526. Walter de Gruyter. find
  3. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308.
  4. Ann Bies, Mark Ferguson, Karen Katz, Robert Mac-Intyre, Victoria Tredinnick, Grace Kim, Mary Ann Marcinkiewicz, and Britta Schasberger. 1995. Bracketing guidelines for Treebank II style Penn Treebank project. find
  5. Yanai Elazar and Yoav Goldberg. 2019. Where’s my head? Definition, data set, and models for numeric fused-head identification and resolution. Transactions of the Association for Computational Linguistics, 7:519–535.
  6. Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English Web Treebank. Technical Report LDC2012T13, Linguistic Data Consortium, Philadelphia, PA. find
  7. Martin Emms. 2008. Tree distance and some other variants of evalb. In Proc. of LREC, Marrakech, Morocco. ↗
  8. E. Black, S. Abney, D. Flickenger, C. Gdaniec, R. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski. 1991. A procedure for quantitatively comparing the syntactic coverage of English grammars. In Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991. find
  9. Tatiana Bladier, Andreas van Cranenburgh, Kilian Evang, Laura Kallmeyer, Robin Möllemann, and Rainer Osswald. 2018. RRGbank: a Role and Reference Grammar corpus of syntactic structures extracted from the Penn Treebank. In Proc. of the 17th International Workshop on Treebanks and Linguistic Theories (TLT 2018), pages 5–16, Oslo, Norway. find
  10. Sabine Brants, Stefanie Dipper, Peter Eisenberg, Silvia Hansen-Schirra, Esther König, Wolfgang Lezius, Christian Rohrer, George Smith, and Hans Uszkoreit. 2004. TIGER: Linguistic interpretation of a German corpus. Research on Language and Computation, 2(4):597–620. ↗
  11. Chris Brew. 2003. The Cambridge Grammar of the English Language, Rodney Huddleston and Geoffrey K. Pullum. Computational Linguistics, 29(1):144– 147.
  12. John Chen and K. Vijay-Shanker. 2000. Automated extraction of TAGs from the Penn Treebank. In Proceedings of the Sixth International Workshop on Parsing Technologies, pages 65–76, Trento, Italy. Association for Computational Linguistics. find
  13. Martin Čmejrek, Jan Cuřín, Jan Hajič, and Jiří Havelka. 2005. Prague Czech-English Dependency Treebank: resource for structure-based MT. In Proc. of EAMT, pages 73–78, Budapest, Hungary. find
  14. Dan Flickinger, Yi Zhang, and Valia Kordoni. 2012. DeepBank. A dynamically annotated treebank of the Wall Street Journal. In Proc. of the 11th International Workshop on Treebanks and Linguistic Theories, pages 85–96. find
  15. Kim Gerdes, Bruno Guillaume, Sylvain Kahane, and Guy Perrier. 2018. SUD or surface-syntactic Universal Dependencies: An annotation scheme near-isomorphic to UD. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 66–74, Brussels, Belgium. Association for Computational Linguistics. ↗
  16. Michael Heilman and Noah A. Smith. 2010. Tree edit models for recognizing textual entailments, paraphrases, and answers to questions. In Proc. of NAACL-HLT, pages 1011–1019, Los Angeles, California.
  17. Julia Hockenmaier and Mark Steedman. 2007. CCGbank: A corpus of CCG derivations and dependency structures extracted from the Penn Treebank. Computational Linguistics, 33(3):355–396. ↗
  18. Rodney Huddleston and Geoffrey K. Pullum. 2002. The Cambridge Grammar of the English Language. Cambridge University Press, Cambridge. ↗
  19. Rodney Huddleston, Geoffrey K. Pullum, and Brett Reynolds. 2021. A Student’s Introduction to English Grammar, 2nd edition. Cambridge University Press. find
  20. Robert T. Kasper. 1989. A flexible interface for linking applications to Penman’s sentence generator. In Speech and Natural Language: Proceedings of a Workshop Held at Philadelphia, Pennsylvania, February 21-23, 1989. ↗
  21. Peter W. Culicover. 2004. The Cambridge Grammar of the English Language (review). Language, 80(1):127–141. find
  22. Wolfgang Maier. 2010. Direct parsing of discontinuous constituents in German. In Proc. of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 58–66, Los Angeles, CA, USA. find
  23. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proc. of LREC, pages 449–454, Genoa, Italy. ↗
  24. Joan Maling. 2000. A simple argument for subject gaps. In Yosef Grodzinsky, Lewis P. Shapiro, and David Swinney, editors, Language and the Brain. Academic Press, San Diego. ↗
  25. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn Treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994. ↗
  26. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. ↗
  27. Yusuke Miyao, Takashi Ninomiya, and Jun’ichi Tsujii. 2004. Corpus-oriented grammar development for acquiring a Head-driven Phrase Structure Grammar from the Penn Treebank. In Proc. of IJCNLP, Lecture Notes in Computer Science, pages 684–693, Hainan Island, China. ↗
  28. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1659–1666, Portorož, Slovenia. European Language Resources Association (ELRA). ↗
  29. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proc. of LREC, pages 4027–4036, Marseille, France. ↗
  30. Stephan Oepen, Kristina Toutanova, Stuart Shieber, Christopher Manning, Dan Flickinger, and Thorsten Brants. 2002. The LinGO Redwoods Treebank: Motivation and Preliminary Applications. In Proc. of COLING. ↗
  31. Mateusz Pawlik and Nikolaus Augsten. 2011. RTED: A robust algorithm for the tree edit distance. arXiv:1201.0230 [cs.DB]. ↗
  32. Mateusz Pawlik and Nikolaus Augsten. 2016. Tree edit distance: Robust and memory-efficient. Information Systems, 56:157–173. ↗
  33. John Payne, Rodney Huddleston, and Geoffrey K. Pullum. 2007. Fusion of functions: The syntax of once, twice and thrice. Journal of Linguistics, 43(3):565– 603. find
  34. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proc. of CoNLL, pages 143–152, Sofia, Bulgaria. find
  35. Geoffrey K. Pullum and Brett Reynolds. 2013. New members of ‘closed classes’ in English. Manuscript. find
  36. Geoffrey K. Pullum and James Rogers. 2008. Expressive power of the syntactic theory implicit in the Cambridge Grammar of the English Language. In Annual Meeting of the Linguistics Association of Great Britain. University of Essex. find
  37. Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2004. Mapping dependencies trees: an application to question answering. In Proc. of the International Symposium on Artificial Intelligence and Mathematics (AIM). find
  38. Brett Reynolds, Aryaman Arora, and Nathan Schneider. 2022. CGELBank: CGEL as a framework for English syntax annotation. arXiv:2210.00394 [cs.CL]. ↗
  39. Brett Reynolds, Nathan Schneider, and Aryaman Arora. 2023. CGELBank annotation manual v1.0. arXiv:2305.17347 [cs.CL]. ↗
  40. Milos Simic. 2022. Tree Edit Distance. Baeldung on Computer Science. Accessed 23 April 2023. find
  41. Sebastian Sulger, Miriam Butt, Tracy Holloway King, Paul Meurer, Tibor Laczkó, György Rákosi, Cheikh Bamba Dione, Helge Dyvik, Victoria Rosén, Koenraad De Smedt, Agnieszka Patejuk, Özlem Çetinoğlu, I Wayan Arka, and Meladel Mistica. 2013. ParGramBank: The ParGram parallel treebank. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 550–560, Sofia, Bulgaria. Association for Computational Linguistics. find
  42. John Torr. 2018. Constraining MGbank: Agreement, L-selection and supertagging in Minimalist Grammars. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 590–600, Melbourne, Australia. Association for Computational Linguistics. ↗
  43. Reut Tsarfaty, Joakim Nivre, and Evelina Andersson. 2011. Evaluating dependency parsing: Robust and heuristics-free cross-annotation evaluation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 385–396, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  44. Andreas van Cranenburgh. 2018. Active DOP: A constituency treebank annotation tool with online learning. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 38–42, Santa Fe, New Mexico. Association for Computational Linguistics. find
  45. Andreas van Cranenburgh, Remko Scha, and Rens Bod. 2016. Data-oriented parsing with discontinuous constituents and function tags. Journal of Language Modelling, 4(1):57–111. ↗
  46. Stephen Wan, Mark Dras, Robert Dale, and Cécile Paris. 2006. Using dependency-based features to take the ‘para-farce’ out of paraphrase. In Proc. of the 2006 Australasian Language Technology Workshop, pages 131–138, Sydney, Australia. find
  47. Kaizhong Zhang and Dennis Shasha. 1989. Simple fast algorithms for the editing distance between trees and related problems. SIAM Journal on Computing, 18(6):1245–1262. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/reynolds-2023-cgel-annotation/