Elamur Research⌕
☰
home / papers / 2017 / Joint Learning of Dialog Act Segmentation and Recognition in Spoken Dialog Using Neural Networks

Joint Learning of Dialog Act Segmentation and Recognition in Spoken Dialog Using Neural Networks

Paper · 2017 · Tianyu Zhao and Tatsuya Kawahara

Summary

Spoken dialog systems work on ASR output or transcripts, which have no punctuation. Before an utterance can be assigned a dialog act (DA) such as Greeting or Statement, it must first be split into DA units. Zhao and Kawahara treat this splitting as word-level sequence labeling with an "IE" scheme (inside / end of segment). DA recognition is then a classification task over whole segments.

For recognition, they build a hierarchical network. A BiLSTM with max pooling turns each segment into a vector, and a second BiLSTM reads the last few segment vectors so that dialog history informs the label. They compare three ways to combine the two tasks:

  • a cascading pipeline,
  • a joint coding model that merges both tag sets into one word-level CRF-style tagger,
  • a joint training model in which the segmenter and the recognizer share word embeddings.

On Japanese spoken dialog data, joint training is best on segmentation, recognition and the combined task. Both hierarchical models clearly beat CRF baselines on the joint task. The work shows that two tasks operating on units of different granularity can still be trained together profitably.

Key points

  • Segmentation (DSER): joint training reaches 9.7% error, against 10.5% for cascading. A CRF that also uses the previous tag (CRF-Bi) does worse on segmentation (11.1%) than a plain n-gram CRF.
  • DA recognition: joint training improves on cascading by 0.8% accuracy, 0.9% Macro F1 and 0.7% Weighted F1.
  • Joint task (DER): CRF scores 36.8% and CRF-Bi 33.5%. The cascading and joint training models lower DER by 4.6 and 6.2 points respectively relative to the CRF baselines.
  • Joint coding model: it gives acceptable segmentation but performs poorly on the joint task. It sees only a ±2-word window (5 words total), so it must commit to a DA type at the start of a segment.
  • Japanese boundaries: segment ends are often signalled by sentence-final words such as "masu", "desu" and "kedo". This is why simple n-gram features already handle segmentation well.

Interesting ideas

  • I1. Segment boundaries can be recovered from unpunctuated text with a two-label inside/end tagging scheme, which turns boundary detection into ordinary sequence labeling.
  • I2. A pragmatic label is predicted from the current unit's encoding plus a short window of previous units; ambiguous lines like "the weather is good" become an Answer or a Statement depending on context.
  • I3. Collapsing a segment-level decision into word-level tags fails because the label has to be chosen before the whole unit has been seen; deciding only after the unit is complete works better.
  • I4. Two tasks at different granularities, word-level and segment-level, can be trained together by sharing only their lowest layer (the embeddings), while each keeps its own architecture.
  • I5. Each task is evaluated separately and with a combined error rate. This revealed that an extra feature (the previous tag) helped recognition but hurt segmentation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): A light IE tagger could split unpunctuated or transcribed input into sentence units before UD parsing, instead of relying on punctuation heuristics.
  • I2 (frames): The speech-act classifier in prag could take the preceding few utterances' frames as context, making the reason for choosing "answer" over "statement" explicit and explainable.
  • I3 (frames): Speech-act and irony decisions should be made once over the finished clause or tree, not incrementally per token.
  • I4 (learn): Tagger and speech-act models could share lexical representations while staying separate, inspectable modules, gaining transfer without merging them into one opaque model.
  • I5 (learn): Evaluation of pipeline changes should report both per-stage and end-to-end metrics, so that a gain at one stage hiding a loss at another gets caught.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Kyoto UniversityKyoto · Japan27 works · 8,049 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Tianyu Zhao and Tatsuya Kawahara. 2017. Joint Learning of Dialog Act Segmentation and Recognition in Spoken Dialog Using Neural Networks. Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zhao2017joint,
  title = {{Joint Learning of Dialog Act Segmentation and Recognition in Spoken Dialog Using Neural Networks}},
  author = {Tianyu Zhao and Tatsuya Kawahara},
  booktitle = {Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2017},
  url = {https://aclanthology.org/I17-1071/}
}

References

14 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse relation language models. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, Harry Bunt, Jan Alexandersson, Jean Carletta, Jae-Woong Choe, Alex Chengyu Fang, Kôiti Hasida, Kiyong Lee, Volha Petukhova, Andrei Popescu-Belis, Laurent Romary, Claudia Soria, and David R. Traum. 2010. Towards an ISO standard for dialogue act annotation. In Nicoletta Calzolari, Khalid Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics 26(3):339–373. 2016. The Association for Computational Linguistics, pages 332–342. ↗
  2. Hamed Khanpour, Nishitha Guntakandla, and Rodney D. Nielsen. 2016. Dialogue act classification in domain-independent conversations using a deep recurrent neural network. In Nicoletta Calzolari, Yuji Matsumoto, and Rashmi Prasad, editors, COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan. ACL, pages 2012–2021.
  3. Diederik P. Kingma and Jimmy Ba. 2014. A method for stochastic optimization. abs/1412.6980. ↗
  4. Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2016. Multi-task cross-lingual sequence tagging from scratch. CoRR abs/1603.06270. ↗
  5. Feifei Zhai, Saloni Potdar, Bing Xiang, and Bowen Zhou. 2017. Neural models for sequence chunking. In Satinder P. Singh and Shaul Markovitch, editors, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA.. AAAI Press, pages 3365–3371. Adam: CoRR Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In Blai Bonet and Sven Koenig, editors, Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA.. AAAI Press, pages 2267–2273. find
  6. Xiaoqing Zheng, Hanyang Chen, and Tianyu Xu. 2013. Deep learning for chinese word segmentation and POS tagging. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. ACL, pages 647–657.
  7. Ji Young Lee and Franck Dernoncourt. 2016. Sequential short-text classification with recurrent and convolutional neural networks. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016. The Association for Computational Linguistics, pages 515–520.
  8. Matthias Zimmermann. 2009. Joint segmentation and classification of dialog acts using conditional random fields. In INTERSPEECH 2009, 10th Annual Conference of the International Speech Communication Association, Brighton, United Kingdom, September 6-10, 2009. ISCA, pages 864–867. find
  9. Wei Li and Yunfang Wu. 2016. Multi-level gated recurrent neural network for dialog act classification. In Nicoletta Calzolari, Yuji Matsumoto, and Rashmi Prasad, editors, COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan. ACL, pages 1970–1979.
  10. Matthias Zimmermann, Yang Liu, Elizabeth Shriberg, and Andreas Stolcke. 2005. Toward joint segmentation and classification of dialog acts in multiparty meetings. In Steve Renals and Samy Bengio, editors, Machine Learning for Multimodal Interaction, Second International Workshop, MLMI 2005, Edinburgh, UK, July 11-13, 2005, Revised Selected Papers. Springer, volume 3869 of Lecture Notes in Computer Science, pages 187–193. find
  11. Nanyun Peng and Mark Dredze. 2016. Improving named entity recognition for chinese social media with word segmentation representation learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 2: Short Papers. The Association for Computer Linguistics. find
  12. Matthias Zimmermann, Andreas Stolcke, and Elizabeth Shriberg. 2006. Joint segmentation and classification of dialog acts in multiparty meetings. In 2006 IEEE International Conference on Acoustics Speech and Signal Processing, ICASSP 2006, Toulouse, France, May 14-19, 2006. IEEE, pages 581–584. find
  13. Silvia Quarteroni, Alexei V. Ivanov, and Giuseppe Riccardi. 2011. Simultaneous dialog act segmentation and classification from human-human spoken conversations. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011, May 22-27, 2011, Prague Congress Center, Prague, Czech Republic. IEEE, pages 5596–5599. find
  14. Elizabeth Shriberg, Andreas Stolcke, Daniel Jurafsky, Noah Coccaro, Marie Meteer, Rebecca Bates, Paul Taylor, Klaus Ries, Rachel Martin, and Carol Van Ess-Dykema. 1998. Can prosody aid the automatic classification of dialog acts in conversational speech? Language and speech 41(3-4):443–492. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2017-joint-learning-dialog-act-segmentation/