Elamur Research⌕
☰
home / papers / 2008 / Annotation of Information Structure: an Evaluation across different Types of Texts

Annotation of Information Structure: an Evaluation across different Types of Texts

Summary

This paper tests how reliably human annotators can apply the LISA guidelines (Linguistic Information Structure Annotation Guidelines) developed within the Potsdam SFB 632 project. The scheme is meant to work across languages and theories. It labels three independent dimensions: information status (given, accessible, new), topic (aboutness vs. frame-setting) and focus (new-information vs. contrastive). It offers core and finer-grained tagsets and uses decision trees and functional tests.

To check reliability, two trained undergraduate annotators labelled three kinds of German material in EXMARaLDA:

  • question/answer pairs and map task dialogues elicited with QUIS,
  • opinion pieces published in newspapers, drawn from the Potsdam Commentary Corpus (PCC).

Agreement was measured with Cohen's κ, computed over NPs and over tokens. Information status proved the most dependable layer. Topic and focus agreement was much lower and depended strongly on text type: the controlled question/answer data scored best and the commentaries worst.

The authors conclude that information-status annotation is feasible. Topic and focus need more training, text-type-specific instructions and sharper rules for focus boundaries. These findings matter for building IS corpora used in discourse analysis and NLP tasks such as anaphora resolution.

Key points

  • The evaluation corpus totals 1,940 tokens: 42 QuAn pairs (573 tokens), 2 map task dialogues (478 tokens) and 5 PCC commentaries (889 tokens).
  • Information status reached κ of about .60–.80 with the core tagset. The extended tagset was slightly but consistently lower (NP-based κ .73 / .61 / .55).
  • Topic agreement was high only for question/answer data (NP-based κ ≈ .91, partial evaluation). It fell to roughly .44–.51 for dialogues and commentaries.
  • Focus was the weakest layer, dropping to κ = .19 (tokens) on PCC. The main difficulty was deciding where a focus span begins.
  • Compared with related work (Nissim et al. 2004, κ = .845; Hempelmann et al. 2005; Paggio 2006 on DanPASS), the authors attribute the gap partly to the short three-day training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Part-of-speech tagging

Cited by

17 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 15 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ritz2008annotation,
  title = {{Annotation of Information Structure: an Evaluation across different Types of Texts}},
  author = {Ritz, Julia and Dipper, Stefanie and Götze, Michael},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  doi = {10.63317/3ni4c3iexg5j},
  url = {https://doi.org/10.63317/3ni4c3iexg5j}
}

References

28 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anne H. Anderson, Miles Bader, Ellen G. Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Carrod, Stephen Isard, Jacqlene Kowtko, Jan McAllister, Jim Miller, Catherine Sotillo, Henry Thompson, and Regina Weinert. 1991. The HCRC Map Task Corpus. Language and Speech, 34:351–366. find
  2. Ron Artstein and Massimo Poesio. 2005. Kappa  = Alpha (or Beta). CSM 437, Department of Computer Science, University of Essex, UK. find
  3. Jean Carletta. 1996. Assessing Agreement on Classification Tasks: The Kappa Statistic. Computational Linguistics, 22(2):249–254. find our card
  4. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37–46. find
  5. Stefanie Dipper, Michael Götze, and Stavros Skopeteas, editors. 2007. Information Structure in Cross-Linguistic Corpora: Annotation Guidelines for Phonology, Morphology, Syntax, Semantics, and Information Structure, volume 7 of Interdisciplinary Studies on Information Structure (ISIS). Universität… find
  6. Jeanette K. Gundel, Nancy Hedberg, and Ron Zacharski. 1993. Cognitive status and the form of referring expressions in discourse. Language, 69(2):279–307. find
  7. Eva Hajic̆ová, Jarmila Panevová, Petr Sgall, Alena Böhmová, Markéta Ceplová, and Veronika R̆eznı́c̆ková. 2000. A Manual for Tectogrammatical Tagging of the Prague Dependency Treebank. TR 2000-09, ÚFAL/CKL, Prague, Czech Republic. find
  8. Nancy Hedberg, Jeanette K. Gundel, and Ron Zacharski. 2007. Directly and Indirectly Anaphoric Demonstrative and Personal Pronouns in Newspaper Articles. In Proceedings of DAARC 2007 (the Sixth Discourse Anaphora and Anaphora Resolution Colloquium), pages 31–36, Lagos, Portugal. find
  9. Christian F. Hempelmann, David Dufty, Philip M. McCarthy, Arthur C. Graesser, Zhiqiang Cai, and Danielle S. McNamara. 2005. Using LSA to automatically identify givenness and newness of noun-phrases in written discourse. In B. Bara, editor, Proceedings of the 27th Annual Meetings of the Cognitive Sci… find
  10. Nobo Komagata, 2001. An Evaluation Method for Identifying Information Structure. Department of Computer and Information Science, University of Pennsylvania. Manuscript. find
  11. Costanza Navarretta. 2004. An algorithm for resolving individual and abstract anaphora in danish texts and dialogues. In Proceedings of the Workshop on Reference Resolution and its Applications: ACL 2004, pages 95– 102, Barcelona, Spain. Association for Computational Linguistics. find our card
  12. Ani Nenkova, Advaith Siddharthan, and Kathleen McKeown. 2005. Automatically learning cognitive status for multi-document summarization of newswire. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 241–248, Vancouver, Bri… find
  13. Malvina Nissim, Shipra Dingare, Jean Carletta, and Mark Steedman. 2004. An Annotation Scheme for Information Status in Dialogue. In Proceedings of the 4th Language Resources and Evaluation Conference (LREC 2004), pages 1023–1026, Lisbon, Portugal. find
  14. Patrizia Paggio. 2006. Annotating Information Structure in a Corpus of Spoken Danish. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006), pages 1606–1609, Genova, Italy. find our card
  15. Massimo Poesio and Renata Vieira. 1998. A Corpus-based Investigation of Definite Description Use. Computational Linguistics, 24(2):183–216. find
  16. Livia Polanyi, Martin van den Berg, and David Ahn. 2003. Discourse Structure and Sentential Information Structure. An Initial Proposal. Journal of Logic, Language and Information, 12(3):337–350. find
  17. Ellen F. Prince. 1981. Toward a Taxonomy of Given-New Information. Radical Pragmatics, pages 223–55. find
  18. David Reitter and Manfred Stede. 2003. Step by step: underspecified markup in incremental rhetorical analysis. In Proc. of the 4th International Workshop on Linguistically Interpreted Corpora (LINC-03), pages 77–84, Budapest, Hungary. find
  19. Arndt Riester, Lorena Killmann, David Lorenz, and Melanie Portz, 2008. Richtlinien zur Annotation von Gegebenheit und Kontrast in Projekt A1 SFB 732 (draft version as of March 2008). University of Stuttgart, Germany. Manuscript. find
  20. Arndt Riester. 2008. A semantic explication of Information Status and the underspecification of the recipients’ knowledge. In A. Gronn, editor, Proceedings of Sinn und Bedeutung 12, Oslo. find
  21. Susanne Salmon-Alt and Renata Vieira. 2002. Nominal Expressions in Multilingual Corpora. In Proceedings of the 3rd International Conference on Language Resources and Evaluation (LREC 2002), pages 1627–1634, Las Palmas, Canary Islands, Spain. find
  22. Thomas Schmidt. 2004. EXMARaLDA - ein System zur computergestützten Diskurstranskription. In Alexander Mehler and Henning Lobin, editors, Automatische Tex-tanalyse - Systeme und Methoden zur Annotation und Analyse natürlichsprachlicher Texte, pages 203–218. Wiesbaden: VS Verlag für Sozialwissensc… find
  23. Petr Sgall, Eva Hajic̆ová, and Jarmila Panevová. 1986. The Meaning of the Sentence in Its Semantic and Pragmatic Aspects. Reidel, Dordrecht. find
  24. Stavros Skopeteas, Ines Fiedler, Sam Hellmuth, Anne Schwarz, Ruben Stoel, Gisbert Fanselow, and Manfred Krifka. 2006. Questionnaire on Information Structure: Reference Manual. Interdisciplinary Studies on Information Structure (ISIS), Vol. 4, Universitätsverlag Potsdam, Potsdam, Germany. find
  25. Jennifer Spenader. 2003. Between binding and accomodation. In Peter Kuehnlein, Hannes Rieser, and Henk Zeevat, editors, Perspectives on Dialogue in the New Millennium, pages 79–110. John Benjamins. find
  26. Manfred Stede. 2004. The Potsdam Commentary Corpus. In Bonnie Webber and Donna K. Byron, editors, Association for Computational Linguistics (ACL) 2004 Workshop on Discourse Annotation, pages 96–102, Barcelona, Spain. find
  27. Michael Strube. 1998. Never Look Back: An Alternative to Centering. In Proceedings of COLING-ACL ’98, pages 1251–1257, Montréal, Québec, Canada. find
  28. Kater̆ina Veselá, Jirı Havelka, and Eva Hajic̆ová. 2004. Annotators’ agreement: The Case of Topic-Focus Articulation. In Proceedings of the 4th Language Resources and Evaluation Conference (LREC 2004), pages 2191– 2194, Lisbon, Portugal. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ritz-2008-annotation-information-structure-evaluation-across/