Elamur Research⌕
☰
home / papers / 2009 / Improving Unsegmented Statistical Dialogue Act Labelling

Improving Unsegmented Statistical Dialogue Act Labelling

Paper · 2009 · Vicent Tamarit, Carlos-D. Martínez-Hinarejos and José Miguel Benedí Ruíz

Summary

Dialogue systems must map each user turn to a sequence of dialogue acts (DAs), where each DA covers a contiguous, non-overlapping span of words called a segment. Most earlier statistical labellers assumed gold segmentation, which a live system never has. Running them on unsegmented turns costs accuracy.

The authors start from an HMM-based labeller. It combines an n-gram over DAs, which can reach back into previous turns, with per-DA HMMs over words. Viterbi decoding then chooses segment boundaries and labels together. They extend it with a factor for the probability of the number of segments r. This factor is obtained through Bayes' rule from a single scalar score per turn, using a Gaussian of the score for each r and priors from turn counts. The score is either the turn length or a sum of probabilities that words or n-grams sit at a segment end.

The best setup uses final bigrams for the score and a 3-gram DA model. It significantly lowers labelling error compared with the unsegmented baseline, but stays well above the error obtained with an oracle r. The method uses only text and is cheap, so it suits typed input or ASR transcripts.

Key points

  • There are two models. The baseline maximises jointly over the segment count and the boundaries with no prior on either. The extended model multiplies in Pr(r|S_c), where p(S_c|r) is a Gaussian fitted per r, p(r) comes from turn counts and p(S_c) is a global Gaussian.
  • Candidate features for the score were:
    • turn length;
    • final words, final bigrams and final trigrams (punctuation such as "?" and "." is a strong end cue);
    • initial words and n-grams;
    • combinations of these.
  • DAER with a 3-gram DA model and 90% confidence intervals of about ±0.5:
    • oracle r: 47.2;
    • length: 54.6;
    • final words: 54.1;
    • final bigrams: 53.5 (best);
    • final trigrams: 54.5.
  • F-measure:
    • baseline: 0.37 (2-gram) and 0.38 (3-gram);
    • oracle r: 0.45 and 0.46;
    • final-bigram estimate: 0.46 and 0.45.
  • Precision stays in a narrow band of 0.44–0.50 across all settings. The authors read this as a sign that the remaining errors come from confusable, somewhat arbitrary DA definitions, not from wrong segment counts.
  • Planned future work includes prosodic and audio features, the task-oriented Dihana corpus, ASR output, and corpus versions with punctuation stripped.

Interesting ideas

  • I1. Rather than committing to a hard segmentation up front, estimate only how many units a span contains and feed that as a soft prior into joint decoding; this is a weaker, easier prediction that still prunes the search.
  • I2. Compress a turn into one interpretable scalar (length or summed boundary-word probabilities) and classify it with per-class Gaussians plus count priors; every decision can be traced to a few numbers.
  • I3. Report an oracle run where the true count is supplied, to measure how much of the error the sub-task could possibly remove.
  • I4. Pair a position-sensitive error rate with position-free precision and recall, so that segmentation mistakes can be told apart from mislabelling.
  • I5. The authors note their boundary cues lean on punctuation and propose retesting on text with marks removed.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): when splitting a sentence into several speech acts, a cheap distribution over the act count could steer the speech-act decoder without forcing brittle boundaries.
  • I2 (tokens): sentence and clause splitting could use a similar transparent boundary-likelihood score whose evidence words can be shown in an explanation.
  • I3 (learn): when evaluating a pipeline stage, run downstream stages with gold input for that stage to bound its potential payoff before investing in it.
  • I4 (learn): speech-act or frame evaluation could report both alignment-aware and bag-of-labels scores to locate where failures arise.
  • I5 (learn): rerunning splitting and tagging on de-punctuated input works as a negative control that exposes over-reliance on surface punctuation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universitat Politècnica de ValènciaValencia · Spain34 works · 3,575 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksSpeech act theory · 229 works

Cite

Cite as Vicent Tamarit, Carlos-D. Martínez-Hinarejos and José Miguel Benedí Ruíz. 2009. Improving Unsegmented Statistical Dialogue Act Labelling. Proceedings of the International Conference RANLP-2009.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{tamarit2009improving,
  title = {{Improving Unsegmented Statistical Dialogue Act Labelling}},
  author = {Vicent Tamarit, Carlos-D. Martínez-Hinarejos and José Miguel Benedí Ruíz},
  booktitle = {Proceedings of the International Conference RANLP-2009},
  year = {2009},
  url = {https://aclanthology.org/R09-1078/}
}

References

17 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Ang, Y. Liu, and E. Shriberg. Automatic dialog act segmentation and classification in multiparty meetings. In Proceedings of the International Conference of Acoustics, Speech, and Signal Processings, volume 1, pages 1061–1064, Philadelphia, 2005. find
  2. J.-M. Benedı́, E. Lleida, A. Varona, M.-J. Castro, I. Galiano, R. Justo, I. López de Letona, and A. Miguel. Design and acquisition of a telephone spontaneous speech dialogue corpus in spanish: Dihana. In Fifth International Conference on Language Resources and Evaluation (LREC), pages 1636–1639, May 2006. find
  3. M. Bisani and H. Ney. Bootstrap estimates for confidence intervals in asr performance evaluation. Acoustics, Speech, and Signal Processing, 2004. Proceedings. (ICASSP ’04). IEEE International Conference on, 1:I–409–12 vol.1, May 2004. find
  4. H. Bunt. Context and dialogue control. THINK Quarterly, 3, 1994. find
  5. M. G. Core and J. F.Allen. Coding dialogues with the damsl annotation scheme. In Fall Symposium on Communicative Action in Humans and Machines, pages 28–35. American Association for Artificial Intelligence, Nov 1997. find
  6. L. Dybkjaer and W. Minker. Recent Trends in Discourse and Dialogue, volume 39 of Text, Speech and Language Technology. Springer, 2008. find
  7. J. Godfrey, E. Holliman, and J. McDaniel. Switchboard: telephone speech corpus for research and development. Acoustics, Speech, and Signal Processing, IEEE International Conference on, 1:517–520, 1992. find
  8. A. Gorin, G. Riccardi, and J. Wright. How may i help you? Speech Communication, 23:113–127, 1997. ↗
  9. D. Jurafsky, E. Shriberg, and D. Biasca. Switchboard swbd-damsl shallow- discourse-function annotation coders manual - draft 13. Technical Report 97-01, University of Colorado Institute of Cognitive Science, 1997. find
  10. L. Levin, K. Ries, A. Thymé-Gobbel, and A. Levie. Tagging of speech acts and dialogue games in Spanish call home. In Workshop: Towards Standards and Tools for Discourse Tagging, pages 42–47, 1999. ↗
  11. C. Martinez-Hinarejos, J. Benedı́, and R. Granell. Statistical framework for a spanish spoken dialogue corpus. Speech Communication, 50:992– 1008, 2008. ↗
  12. C. D. Martı́nez-Hinarejos, R. Granell, and J. M. Benedı́. Segmented and unsegmented dialogue-act annotation with statistical dialogue models. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sesions, pages 563–570, Sydney, Australia,, 17th-21th July 2006.
  13. J. Schatzmann, B. Thomson, and S. Young. Statistical user simulation with a hidden agenda. In Proc. SIGdial Workshop on Discourse and Dialogue, pages 273–282, 2007. find
  14. A. Stolcke, N. Coccaro, R. Bates, P. Taylor, C. van Ess-Dykema, K. Ries, E. Shriberg, D. Jurafsky, R. Martin, and M. Meteer. Dialogue act modelling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):1–34, 2000. find
  15. M. A. Walker. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. Journal of Artificial Intelligence Research, 12:387–416, 2000. ↗
  16. N. Webb, M. Hepple, and Y. Wiks. Dialogue act classification using intra-utterance features. In Proceedings of the AAAI Workshop on Spoken Language Understanding, Pittsburgh, USA, 2005. find
  17. S. Young. Probabilistic methods in spoken dialogue systems. Philosophical Trans Royal Society (Series A), 358(1769):1389–1402, 2000. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tamarit-2009-improving-unsegmented-statistical-dialogue-act/