Elamur Research⌕
☰
home / papers / 2012 / Developing a large semantically annotated corpus

Developing a large semantically annotated corpus

Summary

Resources such as PropBank, FrameNet, the Penn Discourse TreeBank and OntoNotes each cover particular phenomena or stack shallow layers. None of them offers one formally grounded, deep meaning representation for entire texts. The authors introduce the Groningen Meaning Bank (GMB), an English corpus annotated in Discourse Representation Theory (DRT). Its representations are extended with neo-Davidsonian events, VerbNet roles, presuppositions and SDRT-style rhetorical relations, all inside a single DRS per document.

Fully manual gold annotation would be too expensive, so they propose "human-aided machine annotation". The C&C tools and Boxer produce an initial automatic analysis: tokenization, POS, named entities, CCG supertags, parsing and DRS building. Corrections then arrive as stand-off "Bits of Wisdom" (BOWs), small database records of advice. These come from experts using the wiki-like GMB Explorer, from crowd players of the game Wordrobe and, in future, from external tools. A judge component settles conflicts, and documents are reprocessed whenever new BOWs arrive.

A first public-domain release of 1,000 texts is available. The resource matters because it is freely redistributable, works at the document level, and can grow and improve iteratively, much as treebanks did for parsing.

Key points

  • The pipeline runs six steps: tokenization/sentence splitting, POS tagging, NE tagging, CCG supertagging, parsing, and semantic construction with Boxer. The tools are trained on the Penn Treebank and reach up to 98% coverage on newswire.
  • The integrated layers are Penn POS tags, ENE-style named entities, WordNet senses, VerbNet thematic roles, CCG syntax, DRT semantics and SDRT rhetorical relations. All of them are encoded within DRSs.
  • Two BOW types currently exist:
    • boundary(n, level, polarity) marks the presence or absence of a token/sentence boundary.
    • tag(l, r, type, tag) covers POS, sense, NE or supertag.
  • BOWs are applied at two pipeline points: after boundary detection and after tagging. Expert BOWs currently take priority in the judge.
  • Data:
    • Sources are public-domain texts from Voice of America, the CIA Factbook, the open ANC and Aesop's fables.
    • Release 1 contains 1,000 texts, 4,239 sentences and 82,752 tokens.
    • The development version holds 5,000 texts.
    • About 70K more texts (over 1M sentences, 31M tokens) have been collected for later releases.

Interesting ideas

  • I1. Corrections as stored edits: Human fixes are kept as atomic, character-offset stand-off records rather than as edits to the annotation files. They can then be replayed in order after every automatic rerun, so improved tools never erase manual work. This would suit any corpus whose tokenizer or tagger keeps changing.
  • I2. Minimal-change application: Each correction is applied with the smallest change that keeps the annotation well-formed. For example, a sentence boundary also forces a token boundary. A correction that no longer fits, such as one pointing to a vanished token, is simply skipped, which makes the system robust to upstream changes.
  • I3. Leveraging a lexicalised grammar: With CCG, fixing a token's lexical category can be enough to repair the whole parse. Correction effort therefore shifts from trees to word-level labels, which are cheaper and easier to crowdsource.
  • I4. Mixing experts and crowd in one format: Experts and crowd players feed the same database. A game turns a clear majority answer into a correction automatically, and a separate judge decides priority between sources. This pattern of sources with different reliability fits any project combining volunteers and specialists.
  • I5. Copyright-free text selection: Only copyright-free texts are included from the start, so the full annotated corpus can be shared openly. Wiki-style "recent changes" feeds also make collaborative annotation transparent and easy to audit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis · Treebanks · Semantic role labeling

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 57 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{basile2012developing,
  title = {{Developing a large semantically annotated corpus}},
  author = {Basile, Valerio and Bos, Johan and Evang, Kilian and Venhuizen, Noortje},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  doi = {10.63317/3uj8rskf9bkn},
  url = {https://doi.org/10.63317/3uj8rskf9bkn}
}

References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Guillaume Artignan, Mountaz Hascoët, and Mathieu Lafourcade. 2009. Multiscale visual analysis of lexical networks. Information Visualisation, International Conference on, 0:685–690. find
  2. N. Asher and A. Lascarides. 2003. Logics of conversation. Studies in natural language processing. Cambridge University Press. find
  3. Nicholas Asher. 1993. Reference to Abstract Objects in Discourse. Kluwer Academic Publishers. find
  4. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics. Proceedings of the Conference, pages 86–90, Université de Montréal, M… find our card
  5. Valerio Basile, Johan Bos, Kilian Evang, and Noortje Venhuizen. forthcoming 2012. A platform for collaborative semantic annotation. In Proceedings of EACL 2012. find
  6. John Chamberlain, Massimo Poesio, and Udo Kruschwitz. 2008. Addressing the Resource Bottleneck to Create Large-Scale Annotated Texts. In Johan Bos and Rodolfo Delmonte, editors, Semantics in Text Processing. find
  7. STEP 2008 Conference Proceedings, volume 1 of Research in Computational Semantics, pages 375–380. College Publications. find
  8. James Curran, Stephen Clark, and Johan Bos. 2007. Linguistically Motivated Large-Scale NLP with C&C and Boxer. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 33–36, Prague, Czech Republic. find
  9. Christiane Fellbaum, editor. 1998. WordNet. An Electronic Lexical Database. The MIT Press. find
  10. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: the 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 57–60, Stroudsburg, PA, USA. find
  11. Nancy Ide, Christiane Fellbaum, Collin Baker, and Rebecca Passonneau. 2010. The manually annotated sub-corpus: a community resource for and by the people. In Proceedings of the ACL 2010 Conference Short Papers, pages 68–73, Stroudsburg, PA, USA. find
  12. Hans Kamp and Uwe Reyle. 1993. From Discourse to Logic; An Introduction to Modeltheoretic Semantics of Natural Language, Formal Logic and DRT. Kluwer, Dordrecht. find
  13. Karin Kipper, Anna Korhonen, Neville Ryant, and Martha Palmer. 2008. A large-scale classification of English verbs. Language Resources and Evaluation, 42(1):21– 40. find
  14. Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. The Penn Discourse Treebank. In In Proceedings of LREC 2004, pages 2237–2240. find our card
  15. Reinhard Muskens. 1996. Combining Montague Semantics and Discourse Representation. Linguistics and Philosophy, 19:143–186. find
  16. Martha Palmer, Paul Kingsbury, and Daniel Gildea. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. find our card
  17. Rashmi Prasad, Aravind Joshi, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, and Bonnie Webber. 2005. The Penn Discourse TreeBank as a resource for natural language generation. In Proc. of the Corpus Linguistics Workshop on Using Corpora for Natural Language Generation, pages 25–32. find our card
  18. S. Sekine, K. Sudo, and C. Nobata. 2002. Extended named entity hierarchy. In Proceedings of LREC, volume 2. find
  19. Satoshi Sekine. 2010. Sekine’s extended named entity hierarchy. http://nlp.cs.nyu.edu/ene/. ↗
  20. Mark Steedman. 2001. The Syntactic Process. The MIT Press. find
  21. Rob A. van der Sandt. 1992. Presupposition Projection as Anaphora Resolution. Journal of Semantics, 9:333–377. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/basile-2012-developing-large-semantically-annotated-corpus/