📄 Evaluating Dialogue Act Tagging with Naive and Expert Annotators

Summary

This LREC'08 study asks what different kinds of annotators reveal about a dialogue act scheme. Six undergraduates with brief training in pragmatics (naive) and two PhD students with more than two years of experience with the scheme (experts) independently tagged the same pre-segmented Dutch task-oriented dialogues. The dialogues came from OVIS, DIAMOND and a Dutch Map Task collection and mixed human-machine and human-human interaction. The tagset was the multidimensional DIT++ scheme. The authors measured inter-annotator agreement with a taxonomically weighted kappa (κtw) and an ap-ratio, and scored accuracy against a gold standard that they built themselves. They also collapsed DIT++ hierarchies into the coarser LIRICS acts. Experts agreed more and were more accurate. Agreement turned out lower than accuracy, so judging reliability from agreement alone underestimates it. Coarsening the tagset helped naive annotators most. The main lesson is that naive annotators reveal how clear a tagset is, while experts reveal how reliably it can be applied once misunderstanding and tool inexperience are removed.

Key points

• DIT++ has 11 dimensions, including Task, Auto- and Allo-feedback, Turn, Time, Contact and Social Obligations Management. Each dimension allows at most one function per utterance, and functions are either dimension-specific or general-purpose.
• Tagging one utterance took naive annotators 23.2 seconds on average and experts 11.8 seconds.
• On the Task dimension, κtw was 0.56 for naive annotators and 0.82 for experts. Expert agreement was mostly above 0.8.
• After mapping to LIRICS acts, naive κtw rose from 0.56 to 0.65 (Task), 0.36 to 0.71 (auto feedback) and 0.33 to 0.46 (allo feedback). Expert scores rose only from about 0.82 to between 0.85 and 0.88.
• Naive annotators beat experts in two places: agreement on Social Obligations and accuracy on Contact Management. The ap-ratio explains this, because naive coders agreed mainly on the few easy cases they chose to tag.

Interesting ideas

• I1. Use two annotator populations as separate diagnostics: weakly trained coders expose unclear category definitions, and trained coders expose limits that remain even with full understanding.
• I2. Weight disagreements by how close two labels sit in the taxonomy, so confusing sibling acts counts as only a partial error.
• I3. Measure "should anything be tagged in this dimension?" separately from "which label within it?" (the ap-ratio), since mixing the two can make high scores misleading.
• I4. Collapse fine-grained hierarchies into a coarser layer and recompute scores to find which distinctions cause the disagreement; in this paper, feedback levels caused most of it.
• I5. Agreement between coders is not the same as correctness, so pair it with accuracy against a small gold set, and leave out of that gold set the cases where even experts cannot settle the answer.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Treat cheap LLM teacher tiers as "naive" and strong tiers or humans as "expert". Where the cheap tiers disagree, mark labels whose guidelines need sharpening; where the strong tiers disagree, mark genuinely hard cases.
• I2 (frames): Score speech act and semantic role predictions with hierarchy-aware partial credit, so mixing up two kinds of question costs less than mixing up a question and a greeting.
• I3 (frames): For multidimensional pragmatic tagging (feedback, turn, social acts), report detection and classification apart, so a high score on rarely fired dimensions cannot hide missed detections.
• I4 (tree): Re-evaluate the parser and treebank conversion with UD subtypes collapsed (e.g. obl:tmod to obl), to separate core attachment errors from subtype ambiguity.
• I5 (learn): When building silver data from witness parsers that agree, check it against a small human gold set that excludes unresolved cases, so that shared systematic errors do not pass as truth.