Elamur Research⌕
☰
home / papers / 2006 / Measuring annotator agreement in a complex hierarchical dialogue act annotation scheme

Measuring annotator agreement in a complex hierarchical dialogue act annotation scheme

Paper · 2006 · Jeroen Geertzen and Harry Bunt

Summary

The paper asks how to measure reliability when human annotators label dialogue with DIT++, a rich, multidimensional tagset of 86 communicative functions spread over 11 dimensions. Within each dimension the tags often form specificity hierarchies. Plain kappa treats every mismatch as total disagreement, so it undervalues cases where annotators choose related tags. One example is CHECK versus its more specific child POSI-CHECK.

To fix this, the authors propose κ_tw, a variant of Cohen's weighted kappa. Its weights come from the distance between two tags in the taxonomy, so near-misses within one hierarchy count as only partial disagreement.

Six undergraduates who had brief training labelled 1,674 pre-segmented Dutch utterances from task-oriented human-machine and human-human dialogues. The authors computed agreement among three annotators. Most dimensions reach encouraging agreement, and the weighted scores give a fairer picture than unweighted kappa.

The study matters for two reasons. It is the first agreement study for DIT++, and its weighting scheme relies only on taxonomy structure, so it can be reused for any hierarchical label set.

Key points

  • DIT++ has 86 tags in 11 dimensions plus a layer of general-purpose functions. Each utterance may receive at most one tag per dimension.
  • Data came from three Dutch corpora: OVIS (train-connection queries, human-machine), DIAMOND (operating a fax device), and Map Task dialogues (human-human). In total, 1,674 utterances were annotated.
  • The annotation setup:
    • Six students annotated, after about four hours of lectures and a 1-page guideline.
    • Agreement was computed over three annotators using the Davies and Fleiss (1982) multi-rater method.
  • κ_tw scores disagreement by the positions of two tags within the same hierarchy, using the parameters a = 0.75 and β = 0.5.
  • Hierarchies are defined by context-update inclusion: YN-QUESTION ⊂ CHECK ⊂ POSI-CHECK, where each step adds a belief assumption.

Interesting ideas

  • I1. Give partial credit when two labels share an ancestor in a specificity hierarchy instead of counting them as a flat mismatch, because related labels reflect partial agreement.
  • I2. Define "more specific" as "its update operation contains the parent's update," which gives the taxonomy a semantic rather than ad hoc basis.
  • I3. Use an odd number of raters so that a majority usually exists and a single deviant rater can be spotted and down-weighted.
  • I4. When combining per-dimension agreement into one score, weight the dimensions by importance, for example treating task-function disagreement as more serious than disagreement about turn management.
  • I5. Treat the tooling, guidelines and rater experience as confounds, because a tool that hints or enforces consistency changes what the agreement figure actually measures.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Speech-act and frame-role predictions could be scored this way, so that guessing CHECK for POSI-CHECK costs less than guessing APOLOGY.
  • I2 (world): Speech-act types could be modelled as nested sets of belief updates to the text world, so a subtype automatically inherits its parent's effects.
  • I3 (learn): Silver data should use an odd number of witness parsers, with majority voting and explicit flagging of the outlier witness.
  • I4 (answer): Evaluation could weight errors in core predicate-argument frames more heavily than errors in peripheral pragmatic tags.
  • I5 (learn): Evaluation reports should record whether constraints such as "an answer follows a question" were enforced, and should run a negative control without them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tilburg UniversityTilburg · Netherlands50 works · 6,429 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksDecision Sciences · 209 worksStatistics, Probability and Uncertainty · 61 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksSpeech act theory · 229 works

Cite

Cite as Jeroen Geertzen and Harry Bunt. 2006. Measuring annotator agreement in a complex hierarchical dialogue act annotation scheme. Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{geertzen2006measuring,
  title = {{Measuring annotator agreement in a complex hierarchical dialogue act annotation scheme}},
  author = {Jeroen Geertzen and Harry Bunt},
  booktitle = {Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue},
  year = {2006},
  doi = {10.3115/1654595.1654619},
  url = {https://doi.org/10.3115/1654595.1654619}
}

References

23 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James Allen and Mark Core. 1997. Draft of DAMSL: Dialog act markup in several layers. Unpublished manuscript. find
  2. J. Allwood, J. Nivre, and E. Ahlsén. 1993. Manual for coding interaction management. Technical report, Göteborg University. Project report: Semantik och talspråk. find
  3. Harry C. Bunt and Yann Girard. 2005. Designing an open, multidimensional dialogue act taxonomy. In Proceedings of the 9th Workshop on the Semantics and Pragmatics of Dialogue (DIALOR 2005), pages 37–44, Nancy, France, June. find
  4. Harry C. Bunt. 2000. Dialogue pragmatics and context specification. In Harry C. Bunt and William Black, editors, Abduction, Belief and Context in Dialogue; Studies in Computational Pragmatics, pages 81–150. John Benjamins, Amsterdam, The Netherlands. ↗
  5. Harry C. Bunt. 2005. A framework for dialogue act specification. In Joint ISO-ACL Workshop on the Representation and Annotation of Semantic Information, Tilburg, The Netherlands, January. find
  6. Harry C. Bunt. 2006. Dimensions in dialogue annotation. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006), Genova, Italy, May. find
  7. Jean Carletta. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249–254. ↗
  8. Johanneke Caspers. 2000. Pitch accents, boundary tones and turn-taking in dutch map task dialogues. In Proceedings of the 6th International Conference on Spoken Language Processing (ICSLP 2000), volume 1, pages 565–568, Beijing, China. find
  9. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Education and Psychological Measurement, 20:37–46. ↗
  10. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70:213–220. find
  11. Mark G. Core and James F. Allen. 1997. Coding dialogues with the DAMSL annotation scheme. In David Traum, editor, Working Notes: AAAI Fall Symposium on Communicative Action in Humans and Machines, pages 28–35, Menlo Park, CA, USA. American Association for Artificial Intelligence. find
  12. Mark Davies and J.L. Fleiss. 1982. Measuring agreement for multinomial data. Biometrics, 38:1047– 1051. ↗
  13. Barbara Di Eugenio and Michael Glass. 2004. The kappa statistic: a second look. Computational Linguistics, 30(1):95–101. ↗
  14. Barbara Di Eugenio, Pamela W. Jordan, Johanna D. Moore, and Richmond H. Thomason. 1998. An empirical investigation of proposals in collaborative dialogues. In Proceedings of the 17th International Conference on Computational Linguistics and the 36th Annual Meeting of the Association for Computational Linguistics (COLING-ACL 1998), pages 325–329, Montreal, Canada. ↗
  15. Jeroen Geertzen, Yann Girard, Roser Morante, Ielka van der Sluis, Hans Van Dam, Barbara Suijkerbuijk, Rintse van der Werf, and Harry Bunt. 2004. The diamond project (poster,project description). In The 8th Workshop on the Semantics and Pragmatics of Dialogue (Catalog’04). Barcelona, Spain. find
  16. Jeroen Geertzen. 2006. Inter-annotator agreement within dit ++ dimensions. Technical report, Tilburg University, Tilburg, The Netherlands. find
  17. Derek Gross, James F. Allen, and David R. Traum. 1993. The TRAINS 91 dialogues. Technical Report TN92-1, University of Rochester, Rochester, NY, USA. find
  18. Peter A. Heeman and James F. Allen. 1995. The TRAINS 93 dialogues. Technical Report TN94-2, University of Rochester, Rochester, NY, USA. ↗
  19. Klaus Krippendorff. 1980. Content Analysis: An Introduction to its Methodology. Sage Publications, Beverly Hills, CA, USA. find
  20. Stephan Lesch, Thomas Kleinbauer, and Jan Alexandersson. 2005. A new metric for the evaluation of dialog act classification. In Proceedings of the 9th Workshop on the Semantics and Pragmatics of Dialogue (DIALOR 2005), pages 143–146, Nancy, France, june. find
  21. Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue, pages 97–100, Boston, USA, April-May. ↗
  22. Amanda J. Stent. 2000. The monroe corpus. Technical Report TR728/TN99-2, University of Rochester, Rochester, UK. find
  23. Helmer Strik, Albert Russel, Henk van den Heuvel, Catia Cucchiarini, and Lou Boves. 1997. A spoken dialog system for the dutch public transport information service. International Journal of Speech Technology, 2(2):119–129. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/geertzen-2006-measuring-annotator-agreement-complex-hierarchical/