Elamur Research⌕
☰
home / papers / 2017 / On the Effects of Using word2vec Representations in Neural Networks for Dialogue Act Recognition

On the Effects of Using word2vec Representations in Neural Networks for Dialogue Act Recognition

Paper · 2017 · Christophe Cerisara, Pavel Kral and Ladislav Lenc

Summary

The paper looks at dialogue act classification: assigning each already-segmented utterance a functional label such as statement, question, backchannel or agreement. The authors build an LSTM-based deep network that reads only the raw word forms. It uses no part-of-speech tags, speaker information or hand-crafted features. They run it with identical hyper-parameters on English, French and Czech corpora and compare it with a Maximum Entropy baseline. On English the network performs close to the state of the art, and it beats MaxEnt on English and French. The main finding is negative: initializing the network with pretrained word2vec vectors gives no benefit in any language or at any training-set size. The same vectors do help the weaker MaxEnt model. A nearest-neighbour analysis suggests why. Generic word2vec places words like yes and nope close together, while task-trained embeddings keep affirmative and negative words apart. This matters because it shows that popular general-purpose embeddings can encode the wrong similarity for pragmatic tasks.

Key points

  • The model is an LSTM over the words of an utterance, trained end-to-end. Segmentation is assumed to be given, so the task is classification only.
  • The same model and hyper-parameters are used for English, French and Czech, with consistent performance across all three.
  • The DNN beats MaxEnt on English and French but not on Czech. The authors attribute the Czech result to accuracy there already being very high.
  • Pretrained word2vec embeddings help MaxEnt but not the DNN. Preliminary tests with LDA and COALS embeddings did not help either.
  • Cosine similarity of yes to no, not and never:
    • Google News word2vec: 0.392, 0.391 and 0.355 (average 0.380, average rank 71).
    • Task-trained embeddings: 0.292, −0.004 and −0.011 (average 0.092, average rank 318).

Interesting ideas

  • I1. Pull the embedding layer out of a trained model and list the nearest neighbours of a few probe words (yes, what); this shows cheaply and qualitatively what the task actually needs from lexical similarity.
  • I2. Distributional similarity tends to merge opposites like yes/nope, which is harmful whenever polarity carries the meaning.
  • I3. The paper notes that what clusters with other interrogatives in task-trained vectors but with something/anything in word2vec, so interrogative function and semantic content behave as separate dimensions.
  • I4. Running one fixed configuration across several languages, with no retuning, is a simple robustness test against over-tuned hyper-parameters.
  • I5. The finding that extra features help a weak model but not a strong one suggests checking every added resource against both a baseline and the full system.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Probing a few function words in gparse's learned vectors could become a standard diagnostic that shows whether a model separates the distinctions our frames depend on.
  • I2 (frames): Speech-act and answer-polarity detection in prag should depend on explicit lexical polarity lists, not on embedding closeness, so that agreement and refusal are never confused.
  • I3 (tokens): The tagger and lexicon could mark wh-words with an interrogative feature that is kept separate from indefinite pronouns, giving clean cues to later stages.
  • I4 (learn): Evaluations could include a "frozen settings, new treebank" control to check that parser gains do not come from tuning to one corpus.
  • I5 (learn): When adding knowledge sources or silver data, gains should be measured on both the perceptron and gparse parsers before the source is credited.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Laboratoire Lorrain de Recherche en Informatique et ses ApplicationsNancy · France23 works · 3,750 citations🎓University of West Bohemia in PilsenPilsen · Czechia10 works · 38,730 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 3Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 worksLexical semantics & word embeddings · 40 works

Cite

Cite as Christophe Cerisara, Pavel Kral and Ladislav Lenc. 2017. On the Effects of Using word2vec Representations in Neural Networks for Dialogue Act Recognition. Computer Speech and Language, Elsevier, 2018, 47, pp.175 - 193.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1016/j.csl.2017.07.009 · arXiv 2010.11490

@article{cerisara2017effects,
  title = {{On the Effects of Using word2vec Representations in Neural Networks for Dialogue Act Recognition}},
  author = {Christophe Cerisara, Pavel Kral and Ladislav Lenc},
  journal = {Computer Speech and Language, Elsevier, 2018, 47, pp.175 - 193},
  year = {2017},
  doi = {10.1016/j.csl.2017.07.009},
  eprint = {2010.11490},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1016/j.csl.2017.07.009}
}

References

46 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allen, J., Core, M.: Draft of Damsl: Dialog Act Markup in Several Layers. In: http://www.cs.rochester.edu/research/cisd/resources/damsl/RevisedManual/RevisedManual.html (1997) ↗
  2. Ang, J., Liu, Y., Shriberg, E.: Automatic dialog act segmentation and classification in multiparty meetings. In: Proceedings of the Acoustics, Speech, and Signal Processing Conference (ICASSP). vol. 1, pp. 1061– 1064. IEEE (Mar 2005) find
  3. Austin, J.L.: How to do Things with Words. Clarendon Press, Oxford (1962) find
  4. Berger, A.L., Pietra, V.J.D., Pietra, S.A.D.: A maximum entropy approach to natural language processing. Computational linguistics 22(1), 39–71 (1996)
  5. Bunt, H.: Context and Dialogue Control. Think Quarterly 3, 19–31 (May 1994) find
  6. Chollet, F.: Keras. https://github.com/fchollet/keras (2015) ↗
  7. Coop, R., Arel, I.: Mitigation of catastrophic forgetting in recurrent neural networks using a fixed expansion layer. In: Neural Networks (IJCNN), The 2013 International Joint Conference on. pp. 1–7. IEEE (2013) find
  8. Dhillon, R., S., B., Carvey, H., E., S.: Meeting Recorder Project: Dialog Act Labeling Guide. Tech. Rep. TR-04-002, International Computer Science Institute (February 2004) find
  9. Fišel, M.: Machine learning techniques in dialogue act recognition. Eesti Rakenduslingvistika Ühingu aastaraamat 3, 117–134 (2007) find
  10. Fukada, T., Koll, D., Waibel, A., Tanigaki, K.: Probabilistic dialogue act extraction for concept based multilingual translation systems. In: Fifth International Conference on Spoken Language Processing. vol. 6, pp. 2771–2774 (1998) find
  11. Garner, P.N., Browning, S.R., Moore, R.K., Russel, R.J.: A Theory of Word Frequencies and its Application to Dialogue Move Recognition. In: ICSLP’96. vol. 3, pp. 1880–1883. Philadelphia, USA (1996) find
  12. Jeong, M., Lee, G.G.: Triangular-chain conditional random fields. IEEE trans. on Audio, Speech, and Language Processing 16(7), 1287–1302 (Sep 2008) find
  13. Jurafsky, D., Shriberg, E., Biasca, D.: Switchboard SWBD-DAMSL Shallow-Discourse-Function Annotation (Coders Manual, Draft 13). Tech. Rep. 97-01, University of Colorado, Institute of Cognitive Science (1997) find
  14. Jurafsky, D., Bates, R., Coccaro, N., Martin, R., Meteer, M., Ries, K., Shriberg, E., Stolcke, A., Taylor, P., Van Ess-Dykema, C.: Automatic Detection of Discourse Structure for Speech Recognition and Understanding. In: IEEE Workshop on Speech Recognition and Understanding. pp. 88–95. Santa Barbara (1997) find
  15. Kalchbrenner, N., Blunsom, P.: Recurrent convolutional neural networks for discourse compositionality. CoRR abs/1306.3584 (2013), http://arxiv.org/abs/1306.3584 ↗
  16. Keizer, S., R., A., Nijholt, A.: Dialogue Act Recognition with Bayesian Networks for Dutch Dialogues. In: 3rd ACL/SIGdial Workshop on Discourse and Dialogue. pp. 88–94. Philadelphia, USA (July 2002)
  17. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) ↗
  18. Klüwer, T., Uszkoreit, H., Xu, F.: Using syntactic and semantic based relations for dialogue act recognition. In: Proceedings of the 23rd International Conference on Computational Linguistics: Posters. pp. 570– 578. Association for Computational Linguistics (2010)
  19. Konkol, M.: Brainy: A machine learning library. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) Artificial Intelligence and Soft Computing, Lecture Notes in Computer Science, vol. 8468. Springer Berlin Heidelberg (2014) find
  20. Král, P., Cerisara, C.: Automatic dialogue act recognition with syntactic features. Language Resources and Evaluation 48(3), 419–441 (8 February 2014) find
  21. Král, P., Cerisara, C., Klecková, J.: Lexical structure for dialogue act recognition. Journal of Multimedia 2(3), 1–8 (2007) find
  22. Lan, K.C., Ho, K.S., Pong Luk and, R.W., Leong, H.V.: Dialogue act recognition using maximum entropy. Journal of the American Society for Information Science and Technology 59(6), 859–874 (2008), http://dx.doi.org/10.1002/asi.20777 ↗
  23. Levin, L., Langley, C., Lavie, A., Gates, D., Wallace, D., Peterson, K.: Domain Specific Speech Acts for Spoken Language Translation. In: 4th SIGdial Workshop on Discourse and Dialogue. pp. 208–217. Sapporo, Japan (2003) find
  24. Liang, W.B., Wu, C.H., Chen, C.P.: Semantic information and derivation rules for robust dialogue act detection in a spoken dialogue system. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2. pp. 603–608. Association for Computational Linguistics (2011)
  25. Lison, P.: A hybrid approach to dialogue management based on probabilistic rules. Computer Speech & Language 34(1), 232–255 (2015) find
  26. Margolis, A., Livescu, K., Ostendorf, M.: Domain adaptation with unlabeled data for dialog act tagging. In: Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing. pp. 45–52. Association for Computational Linguistics (2010)
  27. Martı́nez-Hinarejos, C.D., Benedı́, J.M., Tamarit, V.: Unsegmented dialogue act annotation and decoding with n-gram transducers. IEEE/ACM Transactions on Audio, Speech, and Language Processing 23(1), 198–211 (2015) find
  28. Petukhova, V., Bunt, H.: Incremental dialogue act understanding. In: Proceedings of the Ninth International Conference on Computational Semantics. pp. 235–244. Association for Computational Linguistics (2011)
  29. Quarteroni, S., Ivanov, A.V., Riccardi, G.: Simultaneous dialog act segmentation and classification from human-human spoken conversations. In: Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on. pp. 5596–5599. IEEE, Prague, Czech Republic (May 2011) find
  30. Rangarajan Sridhar, V.K., Bangalore, S., Narayanan, S.: Combining lexical, syntactic and prosodic cues for improved online dialog act tagging. Comput. Speech Lang. 23(4), 407–422 (Oct 2009), http://dx.doi.org/10.1016/j.csl.2008.12.001 ↗
  31. Ritter, A., Cherry, C., Dolan, B.: Unsupervised modeling of twitter conversations. In: NAACL HLT 2010 - Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference. pp. 172–180 (2010)
  32. Rojas Barahona, L.M., Lorenzo, A., Gardent, C.: Building and Exploiting a Corpus of Dialog Interactions between French Speaking Virtual and Human Agents. In: Chair), N.C.C., Choukri, K., Declerck, T., Doğan, M.U., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S. (eds.) The eighth international conference on Language Resources and Evaluation (LREC). pp. 1428–1435. European Language Resources Association (ELRA), Istanbul, Turkey (May 2012), https://hal.inria.fr/hal-00726721 ↗
  33. Sanchis, E., Castro, M.J.: Dialogue Act Connectionist Detection in a Spoken Dialogue System. In: Second International Conference on Hybrid Intelligent Systems (HIS2002). pp. 644–651. IOS Press, Santiago de Chile, Chile (1-4 December 2002) find
  34. Shriberg, E., Bates, R., Stolcke, A., Taylor, P., Jurafsky, D., Ries, K., Coccaro, N., Martin, R., Meteer, M., Van Ess-Dykema, C.: Language and Speech, Special Double Issue on Prosody and Conversation, vol. 41, chap. Can Prosody Aid the Automatic Classification of Dialog Acts in Conversational Speech?, pp. 439–487 (1998) find
  35. Sridhar, V.K.R., Bangalore, S., Narayanan, S.: Combining lexical, syntactic and prosodic cues for improved online dialog act tagging. Computer Speech & Language 23(4), 407–422 (2009) find
  36. Sridhar, V.K.R., Narayanan, S., Bangalore, S.: Modeling the intonation of discourse segments for improved online dialog act tagging. In: Acoustics, Speech and Signal Processing, 2008. ICASSP 2008. IEEE International Conference on. pp. 5033–5036. IEEE (2008) find
  37. Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., Van Ess-Dykema, C., Meteer, M.: Dialog Act Modeling for Automatic Tagging and Recognition of Conversational Speech. In: Computational Linguistics. vol. 26, pp. 339–373 (2000) find
  38. Stolcke, A., Shriberg, E., Bates, R., Coccaro, N., Jurafsky, D., Martin, R., Meteer, M., Ries, K., Taylor, P., Van Ess-Dykema, C., et al.: Dialog Act Modeling for Conversational Speech. In: AAAI Spring Symp. on Appl. Machine Learning to Discourse Processing. pp. 98–105 (1998) find
  39. Tur, G., Guz, U., Hakkani-Tur, D.: Model adaptation for dialog act tagging. In: Spoken Language Technology Workshop, 2006. IEEE. pp. 94–97. IEEE (2006) find
  40. Vosoughi, S., Roy, D.: Tweet acts: A speech act classifier for twitter. arXiv preprint arXiv:1605.05156 (2016) ↗
  41. Wallis, S.A.: Binomial confidence intervals and contingency tests: mathematical fundamentals and the evaluation of alternative methods. Journal of Quantitative Linguistics 20(3), 178–208 (2013) find
  42. Webb, N., Hepple, M., Wilks, Y.: Error analysis of dialogue act classification. In: Text, Speech and Dialogue. pp. 451–458. Springer (2005) find
  43. Yahya, A.A., Mahmod, R., Ramli, A.R.: Dynamic bayesian networks and variable length genetic algorithm for designing cue-based model for dialogue act recognition. Computer Speech & Language 24(2), 190–218 (2010) find
  44. Zarisheva, E., Scheffler, T.: Dialog act annotation for twitter conversations. In: Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue. pp. 114–123 (2015)
  45. Zhou, Y., Hu, Q., Liu, J., Jia, Y.: Combining heterogeneous deep neural networks with conditional random fields for chinese dialogue act recognition. Neurocomputing 168, 408 – 417 (2015), http://www.sciencedirect.com/science/article/pii/S0925231215007845 ↗
  46. Zimmermann, M., Liu, Y., Shriberg, E., Stolcke, A.: Toward joint segmentation and classification of dialog acts in multiparty meetings. Machine Learning for Multimodal Interaction pp. 187–193 (2006) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cerisara-2020-effects-using-word2vec-representations-neural/