📄 Collaborative Annotation of Dialogue Acts: Application of a New ISO Standard to the Switchboard Corpus

Summary

The paper examines whether the then-new ISO DIS 24617-2 standard for dialogue act annotation can be applied to real transcribed conversation. The authors convert the Switchboard Dialogue Act (SWBD-DA) Corpus from its original SWBD-DAMSL scheme to ISO tags. SWBD-DAMSL assigns a single label to each slash-unit. The ISO scheme instead tags "functional segments" and allows several acts per segment, each placed in a dimension such as turn management or feedback.

The authors grouped the 303 SWBD-DAMSL tags into 60 clusters and mapped most of them to ISO tags automatically. To check the result, they trained classifiers on lexical features. On the ISO-mapped labels, these reached a weighted F1 of about 0.83.

Four tags cannot be converted mechanically because their ISO equivalent depends on context. For these, the authors defined rules keyed on the preceding act and built a graphical tool for manual mapping. The work matters because it moves a widely used dialogue corpus toward an interoperable standard and enables direct comparison of the two schemes.

Key points

• Corpus scale: SWBD-DA contains 1,155 five-minute conversations, about 1.5M word tokens and 223,606 slash-units. All conversations were used, not only the 1,115 standard training dialogues.
• Tag clustering and skew: 303 distinct tags were found and grouped into 60 functions. Statement-non-opinion alone covers 51.79% of tokens, and the top 10 types cover 93.38%.
• ISO DIS 24617-2 design:
  • It defines nine dimensions: Task, Auto-/Allo-Feedback, Turn, Time, Discourse Structuring, Social Obligations, Own and Partner Communication Management.
  • Functional segments may be discontinuous or overlapping.
  • Tags are stored as stand-off annotation.
• Classification on ISO labels (task 2):
  • Weighted F1 was 0.831, more than 10 points above task 1.
  • Inform reached F1 0.93 and Answer 0.775. InitialGoodbye had the highest precision (0.972).
  • Four rare classes scored F1 = 0.
• Manual mapping:
  • Four tags need manual treatment: action-directive (ad), other answers (no), and the accept (aa) and reject (ar) tags.
  • A GUI shows adjacent turns, the previous five turns, or the whole history.
  • Three annotators will each label 115 random files, and agreement will be measured with kappa.

Interesting ideas

• I1. The meaning of a reactive act like "accept" depends on the act it answers: accepting an offer, a thanking or an apology are different ISO acts. Mapping the act by looking back at the previous act resolves this deterministically.
• I2. One utterance can carry several acts in separate dimensions, such as giving the turn while also asking a question. Storing these acts as parallel stand-off layers avoids forcing a single label.
• I3. The conversion is a hybrid. Rules handle the bulk of tags automatically, and humans handle only the context-dependent residue, through a tool that shows exactly the needed context window.
• I4. Training a classifier on the converted labels serves as a sanity check: if the new tags are learnable from simple features, the mapping is probably coherent.
• I5. The authors flag that the most frequent tags include non-communicative labels such as "segment" or "abandoned". This exposes cases where an annotation scheme mixes form-level bookkeeping with function.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): prag could label replies like "okay"/"yeah" by first classifying the antecedent act, giving an explainable rule rather than a lexical guess.
• I2 (frames): speech-act output in prag could be a set of (dimension, function) pairs per segment, so turn-taking and social moves sit beside the main task act.
• I3 (learn): when converting treebanks or silver data, route only the ambiguous categories to an LLM teacher or human, with a minimal context window attached.
• I4 (learn): after any UD dialect or label conversion, train a cheap model on the result and inspect per-class F1, especially zero-F1 classes, as a signal of broken mappings.
• I5 (world): keep structural markers like abandoned or quoted material out of the event/state layer, so the world of a dialogue only receives genuine communicative acts.