Elamur Research⌕
☰
home / papers / 2010 / Annotating Participant Reference in English Spoken Conversation

Annotating Participant Reference in English Spoken Conversation

Paper · 2010 · John Niekrasz and Johanna D. Moore

Summary

In multi-party talk, references to the people in the conversation (I, you, we, first names) are among the most frequent words. They are also hard to interpret. You and we can be generic, plural or singular, can include or exclude the addressee, or can be filler-like with no real referent. Niekrasz and Moore built an annotation procedure for every person-referring expression in meeting speech, aimed at abstractive summarization and information extraction. Their scheme is a simple, non-anaphoric, coreference-style labeling. Annotators decide four things in turn: whether something refers at all, whether it refers to a person, whether the referent is one specific individual, several specific people or underspecified, and, for underspecified cases, whether the speaker, the addressee and others are included. They labeled about 11,000 person references in AMI and ICSI meetings and checked agreement level by level. Specific-versus-vague reference proved highly reliable. The addressing-based attributes were weaker but still usable. The result is a resource for resolving participant reference in dialogue.

Key points

  • The data come from the AMI corpus (scripted design-scenario meetings) and the ICSI corpus (natural workplace meetings). Each meeting has at least four participants and lasts about 45 minutes on average. In total there are roughly 11,000 annotated person references.
  • The labels are:
    • FUNC-NON-PREF and FUNC-FILLER for words that do not refer to a person;
    • PERSON-SINGLE, PERSON-MULTIPLE and PERSON-OTHER (underspecified) for person-referring words;
    • three binary inclusion attributes (speaker, addressee, other) for underspecified referents.
  • First- and second-person pronouns make up 82.0% of person-referring expressions. The 1PP category (we) accounts for 24.6% and 2P (you) for 23.7%.
  • Agreement was measured with Cohen's kappa at each decision level:
    • markable identification: κ=.94;
    • person vs. non-person referentiality: κ=.77;
    • specificity/cardinality: κ=.91, dropping to κ=.75 for we/you only;
    • inclusion attributes: speaker .72, addressee .50, other .66.
  • The scheme extends earlier annotations of you from Switchboard/ICSI (Gupta et al., Frampton et al., Purver et al.) to all person references. It goes beyond the ACE specific/underspecified/generic distinction by using addressing roles.

Interesting ideas

  • I1. Agreement is scored hierarchically: each decision is judged only on items where coders already agreed on the earlier decisions, so a single upstream disagreement does not inflate errors at every later level.
  • I2. Underspecified we/you are not dumped into one "generic" bin; instead three yes/no flags record whether the speaker, the addressee and third parties are included, which is a cheap, compositional way to describe vague group reference.
  • I3. Pronouns are first sorted into person-referring, non-person-referring and filler uses, since chunks like "there we go" or "you've got X" often name no real participant, and confusion clusters around light verbs such as go, get and have.
  • I4. Reported errors show that "we" often wavers between the people present and the organization they belong to, a metonymic group reading worth modeling explicitly.
  • I5. Proper names of participants are identified in a separate first phase, so later reference labels can link to a fixed roster of known people.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When comparing witness parsers or teacher labels, score frame and role agreement only on tokens whose tree or markable decisions already match, so the blame for disagreement is located at the right stage.
  • I2 (world): Entities for vague pronouns could carry speaker/addressee/other inclusion flags instead of a forced single referent, so the world model can stay underspecified yet still answer "does this include the listener?".
  • I3 (frames): Coreference and role assignment could first check whether a pronoun with a light verb is a real argument or a filler, avoiding phantom agents in who-did-what frames.
  • I4 (frames): Coreference for we could offer two candidate referents, the present group and its parent organization, choosing by predicate type (e.g., wanting a product to be original suggests the organization).
  • I5 (world): Building a participant/character roster before resolving pronouns in dialogues or fables gives coreference a closed, explainable set of targets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as John Niekrasz and Johanna D. Moore. 2010. Annotating Participant Reference in English Spoken Conversation. Proceedings of the Fourth Linguistic Annotation Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{niekrasz2010annotating,
  title = {{Annotating Participant Reference in English Spoken Conversation}},
  author = {John Niekrasz and Johanna D. Moore},
  booktitle = {Proceedings of the Fourth Linguistic Annotation Workshop},
  year = {2010},
  url = {https://aclanthology.org/W10-1841/}
}

References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Thorsten Brants and Alex Franz. 2006. Web 1T 5-gram, Version 1. Linguistic Data Consortium. Catalog ID: LDC2006T13. ↗
  2. Lou Burnard, 2007. Reference Guide for the British National Corpus (XML Edition). Research Technologies Service at Oxford University Computing Services. find
  3. Sasha Calhoun, Malvina Nissim, Mark Steedman, and Jason Brenier. 2005. A framework for annotating information structure in discourse. In Proceedings of the ACL Workshop on Frontiers in Corpus Annotation II: Pie in the Sky.
  4. Jean Carletta, Stephen Isard, Anne H. Anderson, Gwyneth Doherty-Sneddon, Amy Isard, and Jacqueline C. Kowtko. 1997. The reliability of a dialogue structure coding scheme. Computational Linguistics, 23(1):13–31. ↗
  5. Herbert H. Clark. 1996. Using Language. Cambridge University Press, Cambridge. find
  6. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37–46. ↗
  7. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The Automatic Content Extraction (ACE) program: Tasks, data, and evaluation. In Proc. LREC.
  8. Matthew Frampton, Raquel Fernndez, Patrick Ehlen, Mario Christoudias, Trevor Darrell, and Stanley Peters. 2009. Who is ”you”? Combining linguistic and gaze features to resolve second-person references in dialogue. In Proc. EACL. find
  9. John J. Godfrey, Edward Holliman, and J. McDaniel. 1992. SWITCHBOARD: Telephone speech corpus for research and development. In Proc. ICASSP, pages 517–520, San Francisco, CA. find
  10. Surabhi Gupta, John Niekrasz, Matthew Purver, and Daniel Jurafsky. 2007a. Resolving “you” in multi-party dialog. In Proc. SIGdial, pages 227–230. find
  11. Surabhi Gupta, Matthew Purver, and Daniel Jurafsky. 2007b. Disambiguating between generic and referential “you” in dialog. In Proc. ACL. find
  12. A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stolcke, and C. Wooters. 2003. The ICSI meeting corpus. In Proc. ICASSP, volume 1, pages 364–367. find
  13. Linguistic Data Consortium, 2008. ACE (Automatic Content Extraction) English Annotation Guidelines for Entities, Version 6.5. Downloaded from http://projects.ldc.upenn.edu/ace/annotation/. ↗
  14. I. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, W. Post, D. Reidsma, and P. Wellner. 2005. The AMI Meeting Corpus. In Proceedings of Measuring Behavior 2005, the 5th International Conference on Methods and Techniques in Behavioral Research, Wageningen, Netherlands. find
  15. Peter Mühlhäusler and Rom Harré. 1990. Pronouns and People: The Linguistic Construction of Social and Personal Identity. Blackwell, Oxford. find
  16. Malvina Nissim, Shipra Dingare, Jean Carletta, and Mark Steedman. 2004. An annotation scheme for information status in dialogue. In Proc. LREC.
  17. R. Passonneau, 1997. Instructions for applying discourse reference annotation for multiple applications (DRAMA). find
  18. Massimo Poesio and Ron Artstein. 2008. Anaphoric annotation in the ARRAU corpus. In Proc. LREC.
  19. Massimo Poesio, 2000. The GNOME Annotation Scheme Manual, Version 4. University of Edinburgh, HCRC and Informatics. find
  20. Massimo Poesio. 2004. Discourse annotation and semantic annotation in the GNOME corpus. In Proceedings of the ACL 2004 Workshop on Discourse Annotation, pages 72–79. find
  21. Matthew Purver, Raquel Fernndez, Matthew Frampton, and Stanley Peters. 2009. Cascaded lexicalised classifiers for second-person reference resolution. In Proc. SIGdial, pages 306–309. find
  22. Stephanie Strassel, Mark Przybocki, Kay Peterson, Zhiyi Song, and Kazuaki Maeda1. 2008. Linguistic resources and evaluation techniques for evaluation of cross-document automatic content extraction. In Proc. LREC. find
  23. Katie Wales. 1996. Personal pronouns in present-day English. Cambridge University Press, Cambridge. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/niekrasz-2010-annotating-participant-reference-english-spoken/