Elamur Research⌕
☰
home / papers / 2006 / Segmented and Unsegmented Dialogue-Act Annotation with Statistical Dialogue Models

Segmented and Unsegmented Dialogue-Act Annotation with Statistical Dialogue Models

Paper · 2006 · Carlos D. Martínez Hinarejos, Ramón Granell and José Miguel Benedí

Summary

Statistical dialogue-strategy models are often evaluated by using them to label turns with dialogue acts (DAs). Earlier work usually assumed two things a live dialogue system lacks: access to the whole dialogue, including future turns, and a correct split of each turn into utterances. The authors build on Stolcke et al.'s N-gram/HMM formulation and propose an "unsegmented model". It uses only the dialogue history up to the current turn and finds utterance boundaries and DA labels together in one Viterbi search. A Word Insertion Penalty (WIP) controls how many segments are produced. They compare it with a "segmented model" that receives gold boundaries. The test data are two very different corpora: SwitchBoard (human–human, open-topic, large vocabulary) and Dihana (human–computer, Spanish train-information task, small vocabulary). Labelling accuracy drops sharply when gold segmentation is removed. The drop is severe on SwitchBoard and milder on Dihana. The main lesson is that segmentation quality strongly limits DA annotation, and therefore limits strategy models built on it.

Key points

  • Model structure:
    • An N-gram over the DA sequence is combined with HMMs that give the likelihood of a word span under each DA.
    • Decoding maximises jointly over the segmentation and the label sequence.
    • The model uses only word transcriptions, with no prosody.
  • SwitchBoard data:
    • 1,155 conversations, 126,754 turns and a vocabulary of 21,797 words.
    • It is annotated with the simplified DAMSL scheme, which has 42 labels.
  • Dihana data:
    • 900 Wizard-of-Oz dialogues with 6,280 user turns and 9,133 system turns.
    • The vocabulary has 812 words, and turns average about 1.5 utterances.
    • It uses a three-level scheme with 248 labels (153 user, 95 system).
  • Best WIP: a value of 50 worked best for the unsegmented setting on both corpora.
  • Typical errors without boundaries:
    • On SwitchBoard, about half of the errors were deleted 'sd'/'sv' utterances, whose length varies widely.
    • On Dihana, the errors were second-level confusions, deleted short "Yes" acceptances, and spurious "Yes" segments inserted after system turns.

Interesting ideas

  • I1. Let the label model choose the boundaries: segmentation is not a separate preprocessing step but is chosen jointly with the labels to maximise their joint probability, which avoids passing early segmentation mistakes down the pipeline.
  • I2. An insertion penalty acts as a single interpretable knob between over-splitting and under-splitting, and it can be tuned on held-out data while inspecting deletion and insertion errors separately.
  • I3. The evaluation is restricted to realistic information, meaning only past context and no gold boundaries, so the score reflects deployment conditions rather than an oracle setup.
  • I4. The authors treat accuracy as a lower bound, because some label or boundary errors would not change the system's reaction; they argue that evaluation should ultimately be based on downstream behaviour.
  • I5. Testing on two deliberately contrasting corpora (open-domain vs task-oriented, large vs small vocabulary) supports claims that a single dataset could not.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Speech-act tagging in prag could score candidate clause/utterance splits jointly with act labels, rather than trusting a fixed sentence splitter.
  • I2 (tokens): A comparable penalty could govern sentence and clause boundary decisions, with insertion and deletion counts reported as separate diagnostics.
  • I3 (learn): Evaluations could report an oracle-input score and a realistic-input score side by side, so the gap shows which upstream stage is the bottleneck.
  • I4 (answer): Errors in frames or the world model could be weighted by whether they actually change the final answer, which separates harmless slips from harmful ones.
  • I5 (learn): Each parser or pragmatics change could be validated on at least two genre-distant text sets, such as fables and school tests, before the conclusions are trusted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universitat Politècnica de ValènciaValencia · Spain34 works · 3,575 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksSpeech act theory · 229 works

Cite

Cite as Carlos D. Martínez Hinarejos, Ramón Granell and José Miguel Benedí. 2006. Segmented and Unsegmented Dialogue-Act Annotation with Statistical Dialogue Models. Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hinarejos2006segmented,
  title = {{Segmented and Unsegmented Dialogue-Act Annotation with Statistical Dialogue Models}},
  author = {Carlos D. Martínez Hinarejos, Ramón Granell and José Miguel Benedí},
  booktitle = {Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions},
  year = {2006},
  doi = {10.3115/1273073.1273146},
  url = {https://doi.org/10.3115/1273073.1273146}
}

References

18 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. N. Alcácer, J. M. Benedı́, F. Blat, R. Granell, C. D. Martı́nez, and F. Torres. 2005. Acquisition and labelling of a spontaneous speech dialogue corpus. In Proceedings of SPECOM, pages 583–586, Patras, Greece. Jan Alexandersson, Bianka Buschbeck-Wolf, Tsutomu Fujinami, Michael Kipp, Stephan Koch, Elis-Utterance Yes, times and fares Yes, I want times and fares of trains that arrive before seven. On thursday in the afternoon Figure 4: An example of errors produced by the model in the Dihana corpus abeth Maier, Norbert Reithinger, Birte Schmitz, and Melanie Siegel. 1998. Dialogue acts in VERBMOBIL-2 (second edition). Technical Report 226, DFKI GmbH, Saarbrücken, Germany, July. find
  2. J. Ang, Y. Liu, and E. Shriberg. 2005. Automatic dialog act segmentation and classification in multiparty meetings. In Proceedings of the International Conference of Acoustics, Speech, and Signal Processings, volume 1, pages 1061–1064, Philadelphia. find
  3. H. Aust, M. Oerder, F. Seide, and V. Steinbiss. 1995. The philips automatic train timetable information system. Speech Communication, 17:249–263. ↗
  4. J. M. Benedı́, A. Varona, and E. Lleida. 2004. Dihana: Dialogue system for information access using spontaneous speech in several environments tic2002-04103-c03. In Reports for Jornadas de Seguimiento - Programa Nacional de Tecnologı́as Informáticas, Málaga, Spain. find
  5. Mark G. Core and James F. Allen. 1997. Coding dialogs with the damsl annotation scheme. In Working Notes of AAAI Fall Symposium on Communicative Action in Humans and Machines, Boston, MA, November. find
  6. Layla Dybkjaer and Niels Ole Bernsen. 2000. The mate workbench. find
  7. N. Fraser, 1997. Assessment of interactive systems, pages 564–614. Mouton de Gruyter. find
  8. J. Godfrey, E. Holliman, and J. McDaniel. 1992. Switchboard: Telephone speech corpus for research and development. In Proc. ICASSP-92, pages 517– 520. find
  9. A. Gorin, G. Riccardi, and J. Wright. 1997. How may i help you? Speech Communication, 23:113–127. ↗
  10. Hilda Hardy, Kirk Baker, Laurence Devillers, Lori Lamel, Sophie Rosset, Tomek Strzalkowski, Cristian Ursu, and Nick Webb. 2002. Multi-layer dialogue annotation for automated multilingual customer service. In Proceedings of the ISLE Workshop on Dialogue Tagging for Multi-Modal Human Computer Interaction, Edinburgh, Scotland, December. find
  11. Hilda Hardy, Tomek Strzalkowski, and Min Wu. 2003. Dialogue management for an automated multilingual call center. In Proceedings of HLT-NAACL 2003 Workshop: Research Directions in Dialogue Processing, pages 10–12, Edmonton, Canada, June. 1st level Acceptance Question Acceptance Question Answer 2nd level Dep Hour,Fare Dep Hour,Fare Dep Hour,Fare Dep Hour,Fare Time D. Jurafsky, E. Shriberg, and D. Biasca. 1997. Switchboard swbd-damsl shallow- discourse-function annotation coders manual - draft 13. Technical Report 97-01, University of Colorado Institute of Cognitive Science. find
  12. J. Van Kuppevelt and R. W. Smith. 2003. Current and New Directions in Discourse and Dialogue, volume 22 of Text, Speech and Language Technology. Springer. find
  13. A. Stolcke, N. Coccaro, R. Bates, P. Taylor, C. van Ess-Dykema, K. Ries, E. Shriberg, D. Jurafsky, R. Martin, and M. Meteer. 2000. Dialogue act modelling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):1– 34. find
  14. Marilyn A. Walker, Diane Litman J., Candace A. Kamm, and Alicia Abella. 1997. PARADISE: A framework for evaluating spoken dialogue agents. In Philip R. Cohen and Wolfgang Wahlster, editors, Proceedings of the Thirty-Fifth Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics, pages 271–280, Somerset, New Jersey. Association for Computational Linguistics. ↗
  15. V. Warnke, R. Kompe, H. Niemann, and E. Nöth. 1997. Integrated Dialog Act Segmentation and Classification using Prosodic Features and Language Models. In Proc. European Conf. on Speech Communication and Technology, volume 1, pages 207–210, Rhodes. find
  16. V. Warnke, S. Harbeck, E. Nöth, H. Niemann, and M. Levit. 1999. Discriminative Estimation of Interpolation Parameters for Language Model Classifiers. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 525–528, Phoenix, AZ, March. find
  17. N. Webb, M. Hepple, and Y. Wilks. 2005. Dialogue act classification using intra-utterance features. In Proceedings of the AAAI Workshop on Spoken Language Understanding, Pittsburgh. find
  18. S. Young. 2000. Probabilistic methods in spoken dialogue systems. Philosophical Trans Royal Society (Series A), 358(1769):1389–1402. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hinarejos-2006-segmented-unsegmented-dialogue-act-annotation/