Elamur Research⌕
☰
home / papers / 2004 / An Annotation Scheme for Information Status in Dialogue

An Annotation Scheme for Information Status in Dialogue

Paper · 2004 · Malvina Nissim, Shipra Dingare, Jean Carletta et al.

Summary

The paper proposes a scheme for labelling the information status (IS) of every noun phrase in spontaneous dialogue. IS reflects what the speaker assumes the hearer already knows. Earlier efforts mostly covered only definites or demonstratives, used flat inventories of many classes, and often reported low or no agreement figures. The authors instead use a coarse three-way split based on Prince: old, mediated (inferrable or generally known) and new. Optional finer subtypes sit beneath old and mediated. Two categories handle markables that should not be judged: non-applicable and not-understood. Markables were extracted automatically from the Penn Treebank parses of Switchboard and annotated with the NITE XML Toolkit, using written guidelines and a precedence decision tree. On held-out dialogues, two annotators reached kappa of about .86 on the top-level categories after training. The scheme was then applied to 147 dialogues, producing what the authors describe as the largest IS-annotated corpus available. Such a corpus supports studies of syntactic choice, anaphora resolution, generation and discourse analysis.

Key points

  • The top-level labels are old, mediated and new, plus non-applicable and not-understood. Old has 6 subtypes (identity, event, general, generic, ident generic, relative). Nine subtypes fall under mediated: aggregation, func value, poss, set, event, situation, part, bound and general. New has no subtypes.
  • Reliability was tested on 3 Switchboard dialogues totalling 1738 NPs, with two annotators:
    • The training dialogue DiaA gave K = .794.
    • DiaB and DiaC gave K = .860 and .857.
    • Agreement on non-applicable alone rose from .614 to .916 and .888 after training.
  • The final corpus covers 147 dialogues and 43,358 sentences. It contains 69,004 valid NPs and 16,324 coreference links.
    • 35,299 NPs are old, 23,816 mediated and 9,889 new.
    • 8,127 markables were marked non-applicable and 160 not-understood.
  • Markables are all NPs plus possessive pronouns. Locative, directional and adverbial NPs, disfluencies and traces are excluded or left untagged.
  • Comparable schemes for definite descriptions report lower agreement, roughly K = .45 to .79 depending on the study and language. These include Poesio & Vieira, Spenader, and Salmon-Alt & Vieira.

Interesting ideas

  • I1. Organise labels as a hierarchy: a small, reliably agreed top layer with optional subtypes underneath, so coarse agreement survives even when fine distinctions are disputed.
  • I2. A separate "non-applicable" bucket catches markables that are artefacts of extraction, such as "course" in "of course", "there" in "there is" and expletive "it".
  • I3. An explicit precedence decision tree resolves cases where several labels fit (e.g. old/identity beats mediated/general), which the annotators credited with much of the agreement gain.
  • I4. Bridging subtypes are anchored in substitution tests and lexical resources: WordNet for part-whole links and FrameNet for situation links.
  • I5. Two old entities joined in a coordination form a new mediated "aggregation" unless that group was mentioned before.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Coreference and bridging output could always commit to old/mediated/new and attach a subtype only when confident, so downstream readers get a dependable coarse signal.
  • I2 (world): Mova could filter such NPs before creating world entities, so idioms and expletives never become referents.
  • I3 (learn): When several witness parsers or LLM teachers give overlapping labels, a fixed priority order could merge them deterministically into silver data.
  • I4 (world): Mova's WordNet meronymy and FrameNet frames could license inferred entities, such as "the door" of a home, giving an explainable reason for each mediated link.
  • I5 (world): Mova could create a fresh group entity for coordinations like "John and Mary", linked to its members, so a later "they" resolves to the group.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom381 works · 80,747 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 654 works

Cite

Cite as Malvina Nissim, Shipra Dingare, Jean Carletta and Mark Steedman. 2004. An Annotation Scheme for Information Status in Dialogue. Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{nissim2004annotation,
  title = {{An Annotation Scheme for Information Status in Dialogue}},
  author = {Malvina Nissim, Shipra Dingare, Jean Carletta and Mark Steedman},
  booktitle = {Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)},
  year = {2004},
  doi = {10.63317/5e8ipnwca99c},
  url = {https://doi.org/10.63317/5e8ipnwca99c}
}

References

25 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Ariel. 1990. Accessing Noun Phrase Antecedents. Routledge, New York. find
  2. C. F. Baker, C. J. Fillmore, and J. B. Lowe. 1998. The Berkeley FrameNet project. In Christian Boitet and Pete Whitelock, editors, Proceedings of COLING-ACL, pages 86–90.
  3. B. Birner and G. Ward. 1998. Information Status and Noncanonical Word Order in English. Cambridge University Press. ↗
  4. J. Carletta, S. Dingare, M. Nissim, and T. Nikitina. 2004. Using the NITE XML Toolkit on the Switchboard Corpus to study syntactic choice: a case study. In Proc. of LREC2004. find
  5. J. Carletta. 1996. Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics, 22(2):249–254. ↗
  6. H. H. Clark. 1975. Bridging. In R. Schank and B. Nash-Webber, editors, Theoretical Issues in Natural Language Processing. The MIT Press, Cambridge, MA. find
  7. S. Davies, M. Poesio, F. Bruneseaux, and L. Romary. 1998. Annotating coreference in dialogues: Proposal for a scheme for MATE, July. http://www.hcrc. ed.ac.uk/˜poesio/anno_manual.html. ↗
  8. M. Eckert and M. Strube. 2001. Dialogue acts, synchronising units and anaphora resolution. Journal of Semantics, 17(1):51–89. find
  9. C. Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA. find
  10. J. Godfrey, E. Holliman, and J. McDaniel. 1992. SWITCHBOARD: Telephone speech corpus for research and development. In Proceedings of ICASSP-92, pages 517–520. find
  11. J. Gundel, N. Hedberg, and R. Zacharski. 1993. Cognitive status and the form of referring expressions in discourse. Language, 69(2):274–307. ↗
  12. J. A. Hawkins. 1978. Definiteness and Indefiniteness. Croom Helm, London. find
  13. L. Hirschman and N. Chinchor. 1997. MUC-7 coreference task definition. In Proc. of MUC-7.
  14. K. Lambrecht. 1994. Information structure and sentence form. Topic, focus, and the mental representation of discourse referents. CUP, Cambridge. ↗
  15. S. Löbner. 1985. Definites. Journal of Semantics, 4:279–326. find
  16. R. Passonneau. 1996. Instructions for applying discourse reference annotation for multiple applications (DRAMA). Unpublished manuscript, December. find
  17. M. Poesio and R. Vieira. 1998. A corpus-based investigation of definite descriptions use. Computational Linguistics, 24(2):183–216. find
  18. M. Poesio. 2000. The GNOME annotation scheme manual. http://www.hcrc. ed.ac.uk/˜gnome/anno_manual.html. ↗
  19. E. F. Prince. 1981. Toward a taxonomy of given-new information. In Peter Cole, editor, Radical Pragmatics. Academic Press, New York. find
  20. E. Prince. 1992. The ZPG letter: subjects, definiteness, and information-status. In S. Thompson and W. Mann, editors, Discourse description: diverse analyses of a fund raising text, pages 295–325. John Benjamins, Philadelphia/Amsterdam. find
  21. S. Salmon-Alt and R. Vieira. 2002. Nominal expressions in multilingual corpora: Definites and demonstratives. In Proc. of LREC2002, 2002, pages 1627–1634. find
  22. J. Spenader. 2001. Between binding and accommodation. In P. Kühnlein, H. Rieser, and H. Zeevat, editors, Proceedings of Bi-Dialog 2001, pages 162–173. find
  23. M. Steedman. 2000. The Syntactic Process. The MIT Press, Cambridge, MA. ↗
  24. E. Vallduvı́. 1992. The Informational Component. Garland, New York. find
  25. T. Wasow. 2002. Postverbal Behavior. CSLI Publications. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nissim-2004-annotation-scheme-information-status-dialogue/