Elamur Research
☰
home / papers / 2020 / Efficient Intent Detection with Dual Sentence Encoders

Efficient Intent Detection with Dual Sentence Encoders

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.nlp4convai-1.5.pdf

Summary

Task-oriented dialogue systems must classify each user utterance into an intent, yet commercial projects often start with only a handful of labelled examples per class. The authors argue that fully fine-tuning LM-pretrained encoders such as BERT is expensive and prone to overfitting in such conditions. Instead, they keep two dual sentence encoders frozen. Both were pretrained on conversational response selection: USE and ConveRT. Only a small MLP classifier is trained on their sentence vectors, used separately or concatenated. They evaluate on HWU 64, CLINC 150 and a newly released banking dataset, BANKING 77, against BERT used as a frozen feature extractor and against fully fine-tuned BERT-Large. The dual-encoder classifiers perform best overall, with the largest margins at 10 or 30 examples per intent. They train in minutes on a laptop CPU and barely react to hyperparameter changes, so scarce labels can be used for training instead of validation. The result makes intent detection cheap to build, and BANKING 77 adds a fine-grained single-domain benchmark.

Key points

  • BANKING 77: 13,083 customer-service queries labelled with 77 fine-grained intents in a single banking domain. Utterances average about 12 words, compared with 7 in HWU 64 (25,716 examples, 64 intents, 21 domains) and 8 in CLINC 150 (23,700 examples, 150 intents, 10 domains).
  • Classifier setup: a single 512-unit ReLU hidden layer with a softmax output, placed on fixed USE, ConveRT or USE+ConveRT encodings. It is trained with SGD (learning rate 0.7, linear decay), heavy dropout of 0.75 and 500 iterations.
  • Few-shot evaluation: training sets are subsampled to 10 or 30 examples per intent, while test sets stay unchanged. The training subsets are released for reproducibility.
  • Model compactness: ConveRT is about 59MB, with 6 Transformer layers and 512-dimensional embeddings, compared with 24 layers and 1024 dimensions in BERT-Large. Frozen BERT features clearly underperform BERT in fine-tuning mode.
  • Released resources: code is at github.com/PolyAI-LDN/polyai-models and the dataset at github.com/PolyAI-LDN/task-specific-datasets. Planned extensions include multilingual transfer and out-of-scope detection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

286 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 279 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.nlp4convai-1.5

@inproceedings{casanueva2020efficient,
  title = {{Efficient Intent Detection with Dual Sentence Encoders}},
  author = {Casanueva, Iñigo and Temčinas, Tadas and Gerz, Daniela and Henderson, Matthew and Vulić, Ivan},
  booktitle = {Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI},
  year = {2020},
  doi = {10.18653/v1/2020.nlp4convai-1.5},
  url = {https://doi.org/10.18653/v1/2020.nlp4convai-1.5}
}

References

30 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Al-Rfou, Marc Pickett, Javier Snaider, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2016. Conversational contextual cues: The case of personalization and history for response ranking. CoRR, abs/1606.00372. — [arXiv:1606.00372](https://arxiv.org/abs/1606.00372) find
  2. Matthew Henderson, Iñigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Ivan Vulić, et al. 2019b. ConveRT: Efficient and accurate conversational representations from transformers. arXiv preprint arXiv:1911.03688. find
  3. Yuki Arase and Jun’ichi Tsujii. 2019. Transfer fine-tuning: A BERT case study. In Proceedings of EMNLP-IJCNLP, pages 5392–5403. find
  4. Daniel Braun, Adrian Hernandez Mendez, Florian Matthes, and Manfred Langen. 2017. Evaluating natural language understanding services for conversational question answering systems. In Proceedings of SIGDIAL, pages 174–185. find
  5. Matthew Henderson, Ivan Vulić, Daniela Gerz, Iñigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019c. Training neural response selection for task-oriented dialogue systems. In Proceedings of ACL, pages 5392–5404. find
  6. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - A large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of EMNLP, pages 5016–5026. find
  7. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. In Proceedings of ICLR, volume abs/1905.01969. — [arXiv:1905.01969](https://arxiv.org/abs/1905.01969) find
  8. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder. CoRR, abs/1803.11175. find our card
  9. Talaat Khalil, Kornel Kiełczewski, Georgios Christos Chouliaras, Amina Keldibek, and Maarten Versteegh. 2019. Cross-lingual intent classification in a low resource industrial setting. In Proceedings of EMNLP-IJCNLP, pages 6418–6423. find
  10. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. CoRR, abs/1910.01108. find
  11. Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of EMNLP-IJCNLP, p… find
  12. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958. find
  13. Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019a. Benchmarking natural language understanding services for building conversational agents. In Proceedings of IWSDS. find
  14. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of ACL, pages 3645–3650. find our card
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  16. Shyam Upadhyay, Manaal Faruqui, Gökhan Tür, Dilek Hakkani-Tür, and Larry P. Heck. 2018. (almost) zero-shot cross-lingual spoken language understanding. In Proceedings of ICASSP, pages 6034–6038. find
  17. Zihan Liu, Jamin Shin, Yan Xu, Genta Indra Winata, Peng Xu, Andrea Madotto, and Pascale Fung. 2019c. Zero-shot cross-lingual dialogue systems with transferable latent variables. In Proceedings of EMNLP-IJCNLP, pages 1297–1303. find
  18. Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of EACL, pages 438–449. find
  19. Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of ICML. find
  20. Jason Williams. 2012. A critical analysis of two statistical spoken dialog systems in public use. In Proceedings of SLT. find
  21. Shikib Mehri, Evgeniia Razumovskaia, Tiancheng Zhao, and Maxine Eskenazi. 2019. Pretraining methods for dialog context representation learning. In Proceedings of ACL, pages 3836–3845. find
  22. Anbang Xu, Zhe Liu, Yufan Guo, Vibha Sinha, and Rama Akkiraju. 2017. A new chatbot for customer service on social media. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 3506–3510. find
  23. Matthew Olson, Abraham J. Wyner, and Richard Berk. 2018. Modern neural networks generalize on small data sets. In Proceedings of NeurIPS, pages 3623– 3632. find
  24. Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernández Ábrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Multilingual universal sentence encoder for semantic retrieval. CoRR, abs/1907.04307. find our card
  25. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237. find
  26. Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8BERT: Quantized 8bit BERT. CoRR, abs/1910.06188. — [arXiv:1910.06188](https://arxiv.org/abs/1910.06188) find
  27. Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? Adapting pre-trained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP, pages 7–14. find
  28. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  29. Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. arXiv preprint arXiv:1909.05855. find
  30. Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing. In Proceedings of NAACL-HLT: Tutorials, pages 15–18. find our card