Elamur Research
☰
home / papers / 2019 / Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog

Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-1380.pdf

Summary

Task-oriented assistants must detect what a user wants (intent) and extract its arguments (slots), but labeling data for every new language is expensive. Research on cross-language transfer for this task had been held back by the lack of multilingual corpora annotated under one scheme. The authors release about 57k utterances covering alarm, reminder and weather requests: roughly 43k in English, plus 8.6k Spanish and 5k Thai utterances. Native speakers translated sampled English utterances, and the translations were annotated with the same guidelines. Using a joint biLSTM-CRF model with self-attention for intent prediction, the authors compare three transfer strategies: machine-translating the English training set, cross-lingual static (XLU) embeddings, and a new approach that reuses a multilingual NMT encoder (Multilingual CoVe) as contextual word representations. Translation is best only in the zero-shot case. Once a few hundred target examples exist, the embedding-based methods do better, and multilingual contextual vectors beat static ones when data is very scarce. Monolingual ELMo still beats every cross-lingual option for Spanish, which shows that transfer methods need further work. The dataset gives a shared benchmark for low-resource dialog NLU.

Key points

  • Dataset splits (train/dev/test): English 30,521/4,181/8,621; Spanish 3,617/1,983/3,043; Thai 2,156/1,235/1,692. It covers 12 intent types and 11 slot types, with datetime shared across domains. Thai items with annotator disagreement were dropped because no bilingual adjudicator was available.
  • The encoder variants are CoVe (low-resource→English), Multilingual CoVe (both translation directions, with language-specific decoder start tokens) and Multilingual CoVe with an added autoencoder objective. All are 2-layer biLSTMs built on fairseq, with 1024-dimensional outputs.
  • The MT encoders were trained on Europarl v7, Paracrawl and newstest data for Spanish, and on IWSLT-based data for Thai.
  • Joint training on English plus target-language data consistently helped across languages, data sizes and embedding types. The choice of embedding mattered little, which suggests the gains come mainly from sharing the biLSTM and CRF layers.
  • The data is released at fb.me/multilingual task oriented data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

144 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 141 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1380

@inproceedings{schuster2019cross,
  title = {{Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog}},
  author = {Schuster, Sebastian and Gupta, Sonal and Shah, Rushin and Lewis, Mike},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1380},
  url = {https://doi.org/10.18653/v1/n19-1380}
}

References

44 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Aly, Kushal Lakhotia, Shicong Zhao, Mrinal Mohit, Barlas Oguz, Abhinav Arora, Sonal Gupta, Christopher Dewan, Stef Nelson-Lindall, and Rushin Shah. 2018. PyText: A seamless path from NLP research to production. ArXiv preprint. find
  2. Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016. Many languages, one parser. Transactions of the Association for Computational Linguistics, 4:431–444. find our card
  3. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. Wit 3 : Web inventory of transcribed and translated talks. In Proceedings of the 16 th Conference of the European Association for Machine Translation (EAMT), Trento, Italy. find
  4. Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. find
  5. Wenhu Chen, Jianshu Chen, Yu Su, Xin Wang, Dong Yu, Xifeng Yan, and William Yang Wang. 2018. XL-NBT: A cross-lingual neural belief tracking framework. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), pages 414–424. find
  6. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. In Proceedings of ICLR 2018. find
  7. Alexis Conneau, Guillaume Lample, Ruty Rinott, Holger Schwenk, Ves Stoyanov, Adina Williams, and Samuel R. Bowman. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), pages 2475–2485. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. ArXiv preprint. find
  9. Timothy Dozat, Peng Qi, and Christopher D. Manning. 2017. Stanford’s graph-based neural dependency parser at the CoNLL 2017 shared task. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. find our card
  10. Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015. Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lan… find
  11. Akiko Eriguchi, Melvin Johnson, Orhan Firat, Hideto Kazawa, and Wolfgang Macherey. 2018. Zero-shot cross-lingual classification using multilingual neural machine translation. ArXiv preprint. find
  12. Murhaf Fares, Andrey Kutuzov, Stephan Oepen, and Erik Velldal. 2017. Word vectors, reuse, and replicability: Towards a community repository of large-text resources. In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 271– 276. find
  13. Jonas Gehring, Michael Auli, David Grangier, and Yann N Dauphin. 2016. A Convolutional Encoder Model for Neural Machine Translation. ArXiv preprint. find
  14. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. ArXiv preprint. find our card
  15. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR 2015. find
  16. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit. find
  17. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAAC… find
  18. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In Proceedings of ICLR 2017. find our card
  19. Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the 11th edition of the Language Resources and Evaluation Conference (LREC 2018), pages 1364–1369. find
  20. Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. In Proceedings of Interspeech 2016. find
  21. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), pages 1412–1421. find
  22. Bryan McCann, James Bradbury, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017). find our card
  23. Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP 2011), pages 62–72. find our card
  24. Grégoire Mesnil, Xiaodong He, Li Deng, and Yoshua Bengio. 2013. Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding. In Proceedings of Interspeech 2013. find
  25. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. ArXiv preprint. find
  26. Nikola Mrkšić, Ivan Vulić, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. 2017. Semantic specialization of distributional word vector spaces using monolingual and cross-lingual constraints. Transactions of the Association for Computational Lingui… find
  27. Miryam de Lhoneux, Johannes Bjerva, Isabelle Augenstein, and Anders Søgaard. 2018. Parameter sharing between dependency parsers for related languages. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), pages 4992–4997. find
  28. Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019. Polyglot contextual representations improve crosslingual transfer. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2019). find
  29. Gurvan L’Hostis, David Grangier, and Michael Auli. 2016. Vocabulary selection strategies for neural machine translation. ArXiv preprint. find
  30. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  31. Ying Lin, Shengqi Yang, Veselin Stoyanov, and Heng Ji. 2018. A multi-lingual multi-task architecture for low-resource sequence labeling. In Proceedings of Barbara Plank and Željko Agić. 2018. Distant supervision from disparate sources for low-resource part-of-speech tagging. In Proceedings of the… find
  32. Katherin Yu, Haoran Li, and Barlas Oguz. 2018a. Multilingual seq2seq training with similarity loss for cross-lingual document classification. In Proceedings of The Third Workshop on Representation Learning for NLP, pages 175–179. find
  33. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2017. A survey of cross-lingual word embedding models. ArXiv preprint. find
  34. Xiaodong Yu, Stephen Mayhew, Mark Sammons, and Dan Roth. 2018b. On the strength of character language models for multilingual named entity recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), pages 3073–3077. find
  35. Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang… find
  36. Daniel Zeman and Philip Resnik. 2008. Cross-language parser adaptation between related languages. In Proceedings of the IJCNLP-08 Workshop on NLP for Less Privileged Languages. find
  37. Karan Singla, Dogan Can, and Shrikanth Narayanan. 2018. A multi-task approach to learning multilingual representations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), pages 214–220. find
  38. Aaron Smith, Bernd Bohnet, and Miryam de Lhoneux. 2018. 82 treebanks, 34 models: Universal Dependency parsing with multi-treebank models. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. find
  39. Raymond Hendy Susanto and Wei Lu. 2017. Neural architectures for multilingual semantic parsing. In Proceedings of the 55th Annual Meeting ofthe Association for Computational Linguistics (ACL 2017), pages 38–44. find
  40. Shyam Upadhyay, Manaal Faruqui, Gokhan Tür, Dilek Hakkani-Tür, and Larry Heck. 2018. (Almost) zero-shot cross-lingual spoken language understanding. In Proceedings of the IEEE ICASSP 2018. find
  41. Ngoc Thang Vu. 2016. Sequential convolutional neural networks for slot filling in spoken language understanding. In Proceedings of Interspeech 2016, pages 3250–3254. find
  42. Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A. Smith, and Jaime Carbonell. 2018. Neural cross-lingual named entity recognition with minimal resources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), pages 369–379. find
  43. Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. In Proceedings of ICLR 2017. find
  44. David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of the First International Conference on Human Language Technology Research. find