Elamur Research⌕
☰
home / papers / 2019 / Adaptation of Hierarchical Structured Models for Speech Act Recognition in Asynchronous Conversation

Adaptation of Hierarchical Structured Models for Speech Act Recognition in Asynchronous Conversation

Paper · 2019 · Tasnim Mohiuddin, Thanh-Tung Nguyen and Shafiq Joty

Summary

Speech act recognition (SAR) labels each sentence in a conversation with its communicative function, such as Question, Suggestion, Response or Polite. Asynchronous conversations like forums and emails have little labeled data, while spoken synchronous corpora such as MRDA are large. The authors tackle this gap in three steps. First, they build an end-to-end hierarchical model: a sentence-level bi-LSTM feeds a conversation-level bi-LSTM, topped by either a Softmax or a CRF layer (H-LSTM and H-LSTM-CRF). Second, they train word embeddings on a large unlabeled conversational corpus. Third, they apply domain adversarial training (Ganin et al., 2016) so the encoder learns features that are useful for classifying acts but do not reveal whether a sentence comes from meetings or from written threads. They evaluate on the forum and email datasets QC3, TA and BC3, and on MRDA. Trained on in-domain data alone, the hierarchical models already set a new state of the art. Adversarial adaptation adds clear further gains. This matters because it shows how to reuse labeled data from a mismatched domain without simply pooling it.

Key points

  • Datasets: QC3, TA and BC3 (forum and email threads) are the targets, and the MRDA meeting corpus is the source. Code, conversational embeddings and data are publicly released.
  • Simply concatenating MRDA with the target data (the "Merge" baseline) or fine-tuning on it works worse than adversarial adaptation, for every model and every dataset.
  • Semi-supervised adaptation (50% of target labels):
    • It gains about 11 F1 points absolute over Merge on QC3 and BC3, and 7–8 points over Fine-tune.
    • It also beats in-domain models trained on 100% of the target labels.
  • Full supervision and no supervision:
    • With full target labels, adaptation adds roughly 3 points over the baselines and 3–4 points over in-domain training.
    • With no target labels, adaptation underperforms the transfer baseline, because the much larger MRDA set dominates training.
  • Neural SCL performed worst, possibly because pivot features do not match across the two domains. Most errors confuse Statement with Response or with Suggestion, the same pairs that troubled human annotators.

Interesting ideas

  • I1. An adversarial domain classifier on the shared encoder pushes features toward domain invariance, which beats naive data pooling when source and target differ in style and vocabulary
  • I2. A two-level encoder (words into sentences, sentences into a conversation) combined with a CRF over act labels captures dependencies between consecutive acts, for example that a Response tends to follow a Question
  • I3. The CRF variant adapts better when target labels are scarce, because tag-transition structure learned on the source domain carries over to the target
  • I4. Learning curves show most of the benefit arrives with the first quarter of target labels, which suggests concentrating annotation effort early
  • I5. A failure check that adapts with zero target labels shows the source domain can overwhelm the task, a useful negative control

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): When training speech-act or implicature classifiers on dialogue-style data, a domain-adversarial head could keep the features usable for written fables and school texts.
  • I2 (frames): The prag crate could score speech-act sequences across a discourse with explicit, inspectable transition weights instead of labeling each sentence in isolation.
  • I3 (learn): Low-resource stages could favor structured output layers whose transition tables are learned on silver data and then reused across genres.
  • I4 (learn): Budgets for the LLM teacher tiers could be planned around such curves, labeling a small in-genre seed before scaling up.
  • I5 (learn): Evaluations should include a "source-only" run to detect cases where borrowed treebank or silver data actually hurts the target genre.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢SalesforceSan Francisco · United States37 works · 15,019 citations🎓Nanyang Technological UniversitySingapore · Singapore73 works · 24,390 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Speech act theory · 160 works

Cite

Cite as Tasnim Mohiuddin, Thanh-Tung Nguyen and Shafiq Joty. 2019. Adaptation of Hierarchical Structured Models for Speech Act Recognition in Asynchronous Conversation. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1134 · arXiv 1904.04021

@inproceedings{mohiuddin2019adaptation,
  title = {{Adaptation of Hierarchical Structured Models for Speech Act Recognition in Asynchronous Conversation}},
  author = {Tasnim Mohiuddin, Thanh-Tung Nguyen and Shafiq Joty},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1134},
  eprint = {1904.04021},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1134}
}

References

34 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein GAN. CoRR, abs/1701.07875. ↗
  2. John Langshaw Austin. 1962. How to do things with words. Harvard University Press. find
  3. Vitor R. Carvalho and William W. Cohen. 2005. On the collective classification of email ”speech acts”. In SIGIR ’05, pages 345–352, New York, NY, USA. find
  4. Kyunghyun Cho, Bart van Merriënboer, Çalar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In EMNLP’14, Doha, Qatar. ↗
  5. Chenhui Chu and Rui Wang. 2018. A survey of domain adaptation for neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1304–1319. Association for Computational Linguistics. ↗
  6. William W. Cohen, Vitor R. Carvalho, and Tom M. Mitchell. 2004. Learning to classify email into “speech acts”. In EMNLP’04, pages 309–316.
  7. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. JMLR, 12:2493–2537.
  8. Matt Crane. 2018. Questionable answers in question answering research: Reproducibility and variability of published results. TACL’18, 6:241–252. ↗
  9. Rajdip Dhillon, Sonali Bhagat, Hannah Carvey, and Elizabeth Shriberg. 2004. Meeting Recorder Project: Dialog Act Labeling Guide. Technical report, ICSI Tech. Report. ↗
  10. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi-olette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. JMLR, 17(1):2096–2030. ↗
  11. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In HLT’11: short papers-Volume 2, pages 42–47. ↗
  12. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In JMLR, Sardinia, Italy. find
  13. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  14. Minwoo Jeong, Chin-Yew Lin, and Gary Geunbae Lee. 2009. Semi-supervised speech act recognition in emails and forums. In EMNLP’09, pages 1250– 1259, Singapore. find
  15. Shafiq Joty, Giuseppe Carenini, and Chin-Yew Lin. 2011. Unsupervised Modeling of Dialog Acts in Asynchronous Conversations. In IJCAI’11, pages 1–130, Barcelona. find
  16. Shafiq Joty and Enamul Hoque. 2016. Speech act modeling of written asynchronous conversations with task-specific embeddings and conditional structured models. In ACL ’16, pages 1746–1756, Berlin, Germany.
  17. Dan Jurafsky, Liz Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL Shallow-Discourse-Function Annotation Coders Manual. Technical report, UC Boulder & SRI International. find
  18. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. In CVSC’13, pages 119–126.
  19. Hamed Khanpour, Nishitha Guntakandla, and Rodney Nielsen. 2016. Dialogue act classification in domain-independent conversations using a deep recurrent neural network. In COLING’16, pages 2012–2021.
  20. Diederik P. Kingma and Jimmy Ba. 2014. A method for stochastic optimization. abs/1412.6980. Adam: CoRR, Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Proceedings of the 28th International Conference on Neural Information Processing Systems -Volume 2, NIPS’15, pages 3294–3302, Cambridge, MA, USA. ↗
  21. Harshit Kumar, Arvind Agarwal, Riddhiman Dasgupta, Sachindra Joshi, and Arun Kumar. 2018. Dialogue act sequence labeling using hierarchical encoder with CRF. In AAAI’18. ↗
  22. John Lafferty, Andrew K. McCallum, and Fernando Pereira. 2001. Conditional Random Fields: Probabilistic models for segmenting and labeling sequence data. In ICML. find
  23. Ji Young Lee and Franck Dernoncourt. 2016. Sequential short-text classification with recurrent and convolutional neural networks. In HLT’16, pages 515– 520.
  24. Annie P Louis and Shay B Cohen. 2015. Conversation trees: A grammar model for topic structure in forums. In ACL ’15. find
  25. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic Regularities in Continuous Space Word Representations. In NAACL-HLT ’13, pages 746–751, Atlanta, GA, USA.
  26. Tatsuro Oya and Giuseppe Carenini. 2014. Extractive summarization and dialogue act modeling on email threads: An integrated probabilistic approach. In SIGDIAL’14, page 133–140, Philadelphia, PA, U.S.A.
  27. Michael J. Paul. 2012. Mixed Membership Markov Models for Unsupervised Conversation Modeling. In EMNLP-CoNLL ’12, pages 94–104, Stroudsburg, PA, USA. find
  28. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP’14, pages 1532–1543, Doha, Qatar.
  29. Ashequl Qadir and Ellen Riloff. 2011. Classifying sentences as speech acts in message board posts. In EMNLP’11, pages 748–758, Edinburgh, Scotland, UK.
  30. A. Ritter, C. Cherry, and B. Dolan. 2010. Unsupervised modeling of twitter conversations. In HLT: NAACL’10, LA, California.
  31. Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In AAAI’16. ↗
  32. Maryam Tavafi, Yashar Mehdad, Shafiq Joty, Giuseppe Carenini, and Raymond Ng. 2013. Dialogue act recognition in synchronous and asynchronous conversations. In SIGDIAL’13, Metz, France. ↗
  33. Jan Ulrich, Gabriel Murray, and Giuseppe Carenini. 2008. A publicly available annotated corpus for supervised email summarization. In AAAI’08, Chicago, USA. find
  34. Yftah Ziser and Roi Reichart. 2017. Neural structural correspondence learning for domain adaptation. In CoNLL 2017, pages 400–410. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohiuddin-2019-adaptation-hierarchical-structured-models-speech/