Elamur Research⌕
☰
home / papers / 1997 / The Reliability of a Dialogue Structure Coding Scheme

The Reliability of a Dialogue Structure Coding Scheme

Summary

Dialogue research has long relied on isolated, often invented examples. Annotating large corpora with dialogue structure could give findings firmer empirical grounding, but only if trained coders apply the categories consistently. The authors present a three-tier annotation scheme built for the HCRC Map Task Corpus, a collection of spontaneous, task-oriented spoken dialogues. In each dialogue, a route giver guides a follower across a map with roughly fifteen landmarks.

  • Transactions are the top tier: each covers one major step of the task plan.
  • Conversational games sit in the middle: each runs from an initiation until its purpose is met or dropped, and games may nest.
  • Moves are the bottom tier: initiations and responses labelled by purpose through a decision tree.

Compared with earlier schemes, this one depends less on the specific task, models structure above the utterance level, and is meant for general use alongside other annotation layers such as intonation. The paper then measures inter-coder reliability, using the kappa statistic as indicated by its references. The provided excerpt omits the actual reliability figures.

Key points

  • The move tier has 12 categories:
    • initiations: INSTRUCT, EXPLAIN, CHECK, ALIGN, QUERY-YN, QUERY-W;
    • responses: ACKNOWLEDGE, REPLY-Y, REPLY-N, REPLY-W, CLARIFY;
    • preparation: READY.
  • The tiers loosely mirror the middle levels of Sinclair and Coulthard (1975), and the move set extends Houghton's (1986) interaction frames.
  • Two moves are separated by what the speaker seeks. CHECK asks for confirmation of something the speaker inferred, while ALIGN tests the partner's attention or readiness to proceed.
  • Earlier schemes were more task-bound, for example VERBMOBIL's 17 speech acts and the 9 illocutionary categories of Nagata and Morimoto.
  • The full scheme is documented in the HCRC dialogue structure coding manual (HCRC/TR-82, 1996). The kappa numbers do not appear in this excerpt and are not reported here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

81 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 68 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@article{carletta1997reliability,
  title = {{The Reliability of a Dialogue Structure Coding Scheme}},
  author = {Carletta, Jean and Isard, Amy and Isard, Stephen and Kowtko, Jacqueline C. and Doherty-Sneddon, Gwyneth and Anderson, Anne H.},
  journal = {Computational Linguistics},
  year = {1997},
  doi = {10.5555/972684.972686},
  url = {https://doi.org/10.5555/972684.972686}
}

References

27 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahrenberg, Lars, Nils Dahlb/ick, and Arne J6nsson. 1995. Coding schemes for studies of natural language dialogue. In Proceedings of the AAAI 1995 Spring Symposium on Empirical Methods in Discourse Interpretation and Generation, pages 8-13, March. find
  2. Alexandersson, Jan, Elisabeth Maier, and Norbert Reithinger. 1995. A robust and efficient three-layered dialogue component for a speech-to-speech translation system. In Proceedings of the Seventh European Meeting of the Association for Computational Linguistics, pages 188-193. find
  3. Anderson, Anne H., Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, Catherine Sotillo, Henry Thompson, and Regina Weinert. 1991. The HCRC Map Task Corpus. Language and Speech, 34(4):351-366. find
  4. Bird, Stuart, Sue Browning, Roger Moore, and Martin Russell. 1995. Dialogue move recognition using topic spotting techniques. In ESCA Workshop on Spoken Dialogue Systems--Theories and Applications, pages 45-48, May. find
  5. Carletta, Jean. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249-254. find our card
  6. Carletta, Jean, Amy Isard, Stephen Isard, Jacqueline Kowtko, Gwyneth Doherty-Sneddon, and Anne Anderson. 1996. HCRC dialogue structure coding manual. Technical Report HCRC/TR-82, Human Communication Research Centre, University of Edinburgh, June. find
  7. Carlson, Lauri. 1983. Dialogue Games: An Approach to Discourse Analysis. D. Reidel. find
  8. Clark, Herbert and Edward Schaefer. 1989. Contributing to discourse. Cognitive Science, 13:259-294. find
  9. Condon, Sherri L. and Claude G. Cech. 1996. Functional comparison of face-to-face and computer-mediated decision-making interactions. In Susan C. Herring, editor, Computer-Mediated Conversation: Linguistic, Social and Cross-Cultural Perspectives, pages 65-80. John Benjamins. find
  10. Doherty-Sneddon, Gwyneth, Anne H. Anderson, Claire O'Malley, Steve Langton, Simon Garrod, and Vicki Bruce. Forthcoming. Face-to-face and video mediated communication: A comparison of dialogue structure and task performance. Human Communication Research Centre. find
  11. Greene, John O. and Joseph N. Cappella. 1986. Cognition and talk: The relationship of semantic units to temporal patterns of fluency in spontaneous speech. Language and Speech, 29(2):141-157. find
  12. Grice, Martine, Ralf Benzrnuller, Michelina Savino, and Bistra Andreeva. 1995. The intonation of queries and checks across languages: Data from Map Task dialogues. In Proceedings of the Thirteenth International Congress of Phonetic Sciences, volume 3, pages 648-651, August. find
  13. Grice, Martine and Michelina Savino. 1995. Intonation and communicative function in a regional variety of Italian. In Phonus 1, pages 19-32. Institute of Phonetics, University of the Saarland. find
  14. Grosz, Barbara and Candace Sidner. 1986. Attentions, intentions, and the structure of discourse. Computational Linguistics, 12(3):175-204. find
  15. Houghton, George. 1986. The Production of Language in Dialogue: A Computational Model. Ph.D. thesis, University of Sussex, April. find
  16. Kowtko, Jacqueline C. 1995. The function of intonation in spontaneous and read dialogue. In Proceedings of the XIIIth International Congress of Phonetic Sciences, volume 2, pages 286-289. find
  17. Krippendorff, Klaus. 1980. Content Analysis: An introduction to its Methodology. Sage Publications. find
  18. Landis, J. R. and G. G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33:159-174. find
  19. Lewin, Ian, Martin Russell, David Carter, Sue Browning, Keith Ponting, and Stephen Pulman. 1993. A speech based route enquiry system built from general purpose components. In Proceedings of the 3rd European Conference on Speech Communication and Technology (EUROSPEECH-93), September. find
  20. Merrison, Andrew John, Anne H. Anderson, and Gwyneth Doherty-Sneddon. 1994. An investigation into the communicative abilities of aphasic subjects in task oriented dialogue. Technical Report RP-50, Human Communication Research Centre, June. find
  21. Nagata, Masaaki and Tsuyoshi Morimoto. 1993. An experimental statistical dialogue model to predict the speech act type of the next utterance. In Katsuhiko Shirai, Tetsunori Kobayashi, and Yasunari Harada, editors, Proceedings of the International Symposium on Spoken Dialogue, pages 83--86. find
  22. Newlands, Alison, Anne H. Anderson, and Jim Mullin. 1996. Dialog structure and cooperative task performance in two CSCW environments. In J. Connolly, editor, Linguistic Concepts and Methods in CSCW. Springer-Verlag, chapter 5, pages 41--60. find
  23. Power, Richard J. D. 1979. The organisation of purposeful dialogues. Linguistics, 17:107-152. find
  24. Siegel, Sidney and N. J. Castellan, Junior. 1988. Nonparametric Statistics for the Behavioral Sciences, second edition. McGraw-Hill. find
  25. Sinclair, John M. and R. Malcolm Coulthard. 1975. Towards an Analysis of Discourse: The English Used by Teachers and Pupils. Oxford University Press. find
  26. Sutton, Stephen, Brian Hansen, Terri Lander, David G. Novick, and Ronald Cole. 1995. Evaluating the effectiveness of dialogue for an automated spoken questionnaire. In Proceedings of the AAAI 1995 Spring Symposium on Empirical Methods in Discourse Interpretation and Generation, pages 156-161, March. find
  27. Walker, Marilyn and Steve Whittaker. 1990. Mixed initiative in dialogue: An investigation into discourse segmentation. In Proceedings of the 28th Annual Meeting, Association for Computational Linguistics, pages 70-78. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/carletta-1997-reliability-dialogue-structure-coding-scheme/