Elamur Research⌕
☰
home / papers / 2006 / Backoff Model Training using Partially Observed Data: Application to Dialog Act Tagging

Backoff Model Training using Partially Observed Data: Application to Dialog Act Tagging

Paper · 2006 · Gang Ji and Jeff Bilmes

Summary

The paper addresses dialog act (DA) tagging, where each sentence in a meeting gets a label such as statement, question, or back-channel. Standard taggers assume one word distribution per DA across the whole sentence. The authors argue that utterances pass through DA-specific internal phases, but these sub-DA states are never annotated.

They add a hidden state layer to a DBN tagger, so that words are predicted from the previous word, the DA, and the hidden sub-DA state. Kneser-Ney backoff models need integer counts of fully observed data, so the authors build an embedded training loop:

  • EM learns the state-transition table.
  • Viterbi chooses the most likely state sequence.
  • A smoothed backoff word model is retrained on that hard sequence.

They call the result hidden backoff models (HBMs). On the ICSI MRDA corpus, training likelihood rises at every iteration and test error falls. With per-DA state counts, error drops 6.1% relative to the baseline. This beats earlier prosody-based results and matches the authors' own prosody model without using any acoustics.

Key points

  • Data and toolkit: the MRDA collection of ICSI meeting recordings annotated with dialog acts, with sentence boundaries assumed known. All turns of a meeting are joined into one stream so that a DA bigram can link sentences.
  • Baseline: DA-specific word bigrams built with SRILM, modified Kneser-Ney smoothing, and factored/generalized backoff, decoded with Viterbi.
  • Best configuration: different state counts per DA, namely three for questions, two for statement-like tags, and one for back-channel-like tags. This gives a 6.1% relative error reduction. Using three states for every DA was worse than using two for every DA.
  • Training behaviour: the log likelihood rose strictly at each iteration. Both likelihood and error leveled off after about four iterations, with no visible overfitting.
  • Prosody model and combination:
    • The authors built a prosody model from per-speaker normalized F0 slopes, with one regression per word quantized into 20 bins. It reached 18.4% error, 6.6% better than baseline and statistically tied with the best HBM.
    • Combining prosody with the HBM brought no further gain. The errors fixed by the two methods overlap by more than 50%.

Interesting ideas

  • I1. When a smoothing method requires integer counts and observed variables, alternate soft EM for the latent structure with a hard Viterbi decode, then retrain the count-based smoothed model on that decode; this keeps strong smoothing while still learning hidden variables.
  • I2. Within-sentence structure is modeled as a DA-specific sequence of unlabeled sub-states, so each act type gets its own small internal automaton instead of one flat word distribution.
  • I3. The number of hidden states is set per class according to how complex that class typically is, and giving every class the same capacity proved worse.
  • I4. Before claiming two methods are complementary, measure how much their corrected errors overlap; here an overlap above 50% explained why combining them added nothing.
  • I5. The dominant errors come from very short ambiguous utterances like "yeah", which can be a back-channel, an agreement, or a question; the authors propose forking into short versus long sub-models and conditioning on the previous act.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver-data training could use this loop, letting a latent label such as a sense or clause phase be decoded hard and then fed into count-based, backoff-smoothed component models.
  • I2 (frames): Speech-act recognition in prag could give each act type its own short positional state chain over the UD tree's linear order, keeping the decision explainable as a path of states.
  • I3 (learn): Per-category model capacity, for example per frame or per verb class, should be tuned separately and checked against a uniform-capacity control.
  • I4 (learn): Evaluation of witness parsers or added features should report error-overlap sets, so that redundant signals are not stacked in the hope of additive gains.
  • I5 (frames): Single-word replies could be routed to a dedicated branch that uses the preceding dialog act as context, rather than being forced through the full-clause frame analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States353 works · 200,043 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksSpeech act theory · 229 works

Cite

Cite as Gang Ji and Jeff Bilmes. 2006. Backoff Model Training using Partially Observed Data: Application to Dialog Act Tagging. Proceedings of the Human Language Technology Conference of the NAACL, Main Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ji2006backoff,
  title = {{Backoff Model Training using Partially Observed Data: Application to Dialog Act Tagging}},
  author = {Gang Ji and Jeff Bilmes},
  booktitle = {Proceedings of the Human Language Technology Conference of the NAACL, Main Conference},
  year = {2006},
  doi = {10.3115/1220835.1220871},
  url = {https://doi.org/10.3115/1220835.1220871}
}

References

32 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Ang et al. 2005. Automatic dialog act segmentation and classification in multiparty meetings. In ICASSP. find
  2. P. Bartlett et al. 2004. Exponentiated gradient algorithms for large-margin structured classification. In NIPS. find
  3. S. Bhagat et al. 2003. Labeling guide for dialog act tags in the meeting recordering meetings. Technical Report 2, International Computer Science Insititute. find
  4. J. Bilmes and K. Kirchhoff. 2003. Factored language models and generalized parallel backoff. In Human Lang. Tech., North American Chapter of Asssoc. Comp. Ling., Edmonton, Alberta, May/June. find
  5. J. Bilmes and G. Zweig. 2002. The Graphical Models Toolkit: An open source software system for speech and time-series processing. Proc. IEEE Intl. Conf. on Acoustics, Speech, and Signal Processing. find
  6. S. Chen and J. Goodman. 1998. An empirical study of smoothing techniques for language modeling. Technical report, Computer Science Group, Harvard University. ↗
  7. R. Durbin et al. 1999. Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids. Cambridge University Press. find
  8. D. Gildea and D. Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288. ↗
  9. Y. He and S. Young. 2003. A data-driven spoken language understanding system. In Proc. IEEE Workshop on Automatic Speech Recognition and Understanding, pages 583–588. find
  10. G. Ji and J. Bilmes. 2005. Dialog act tagging using graphical models. In Proc. IEEE Intl. Conf. on Acoustics, Speech, and Signal Processing, Philadelphia, PA, March. find
  11. D. Jurafsky et al. 1997a. Automatic detection of discourse structure for speech recognition and understanding. In Proc. IEEE Workshop on Speech Recognition and Understanding. ↗
  12. D. Jurafsky et al. 1997b. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual. Technical Report 97-02, Institute of Cognitive Science, University of Colorado. find
  13. J. Lafferty et al. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML. find
  14. K. Lee et al. 1997. Restricted representation of phrase structure grammar for building a tree annotated corpus of Korean. Natural Language Engineering, 3(2-3):215–230. ↗
  15. H. Lee et al. 1998. Speech act analysis model of Korean utterances for automatic dialog translation. J. KISS(B) (Software and Applications), 25(10):1443–1452. find
  16. J. Lember and A. Koloydenko. 2004. Adjusted viterbi training. a proof of concept. In Submission. find
  17. K. Ma et al. 2000. Bi-modal sentence structure for language modeling. Speech Communication, 31(1):51–67. find
  18. M. Mast et al. 1996. Automatic classification of dialog acts with semantic classification trees and polygrams. Connectionist, Statistical and Symbolic Approaches to Learning for Natural Language Processing, pages 217–229. find
  19. N. Morgan and H. Bourlard. 1990. Continuous speech recognition using multilayer perceptrons with hidden Markov models. In ICASSP, pages 413–416. find
  20. K. Murphy. 2002. Dynamic Bayesian Networks, Representation, Inference, and Learning. Ph.D. thesis, MIT, Dept. Computer Science. find
  21. R. Neal and G. Hinton. 1998. A view of the EM algorithm that justifies incremental, sparse, and other variants. In Learning in Graphical Models, pages 355–368. Dordrecht: Kluwer Academic Publishers. ↗
  22. R. Pieraccini and E. Levin. 1991. Stochastic representation of semantic structure for speech understanding. In Eurospeech, volume 2, pages 383–386. ↗
  23. N. Reithinger and M. Klesen. 1997. Dialogue act classification using language models. In Eurospeech. find
  24. N. Reithinger et al. 1996. Predicting dialogue acts for a speech-to-speech translation system. In ICLSP, pages 654–657. ↗
  25. J. Searle. 1969. Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press. ↗
  26. E. Shriberg et al. 1998. Can prosody aid the automatic classification of dialog acts in conversational speech? Language and Speech, 41(3–4):439–487. ↗
  27. E. Shriberg et al. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In Proc. of the 5th SIGdial Workshop on Discourse and Dialogue, pages 97–100. ↗
  28. A. Stolcke et al. 1998. Dialog act modeling for conversational speech. In Proc. AAAI Spring Symp. on Appl. Machine Learning to Discourse Processing, pages 98–105. find
  29. A. Stolcke. 2002. SRILM – an extensible language modeling toolkit. In ICLSP, volume 2, pages 901–904. find
  30. C. Sutton et al. 2004. Dynamic conditional random fields: factorized probabilistic models for labeling and segmenting sequence data. In ICML. find
  31. D. Talkin. 1995. A robust algorithm for pitch tracking (rapt). In W. B. Kleijn and K.K. Paliwal, editors, Speech Coding and Synthesis, pages 495–518. Elsevier Science. find
  32. A. Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Trans. on Information Theory, 13(2):260–269. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ji-2006-backoff-model-training-partially-observed/