Elamur Research⌕
☰
home / papers / 2020 / End-to-End Slot Alignment and Recognition for Cross-Lingual NLU

End-to-End Slot Alignment and Recognition for Cross-Lingual NLU

Summary

Goal-oriented dialog systems depend on NLU, which classifies a user's intent and labels slots. Porting such a system to a new language usually means machine-translating the English training data and then projecting slot labels onto the translations. That projection step is error-prone, particularly for languages that differ structurally from English. The authors address this in two ways. First, they build MultiATIS++, an extension of Multilingual ATIS in which professional native translators translated and slot-annotated the English data. The corpus covers nine languages from four families. Second, they propose an end-to-end soft-alignment model on top of multilingual BERT. An attention module learns to link target-language tokens to the source slot labels, and it is trained jointly with intent classification, slot filling and a reconstruction loss, so no external aligner is required. In zero-shot transfer, the model beats MT+fast-align on most languages. It roughly matches the stronger feature-based projection method MT+TMP while needing half the training time, and it is clearly more robust on Turkish. The corpus is publicly released.

Key points

  • MultiATIS++: English, Spanish, German, French, Portuguese, Hindi, Chinese, Japanese and Turkish. It contains 37,084 training and 7,859 test examples, with 18 intents and 84 slot types; the Hindi and Turkish portions are smaller.
  • The soft-alignment model outperforms BERT+MT+fast-align on five of eight target languages. It matches or exceeds MT+TMP on intent accuracy for all languages and on slot F1 for four of them.
  • On Turkish, the model beats MT+TMP by 16.9% slot F1. Hard projection accuracy there is only 20% with fast-align and 39% with TMP, compared with over 70% for the other languages.
  • In the ablation, average slot F1 is 80.00. It falls by 3.6% without the reconstruction loss and by 8.4% without joint training on the source data, while intent accuracy barely changes.
  • In few-shot experiments with a few hundred target examples, soft-align equals (French) or surpasses (Chinese) BERT+MT+TMP on slot F1.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 77 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.410

@inproceedings{xu2020end,
  title = {{End-to-End Slot Alignment and Recognition for Cross-Lingual NLU}},
  author = {Xu, Weijia and Haider, Batool and Mansour, Saab},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.410},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.410}
}

References

35 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. 2019. Multi-source cross-lingual model transfer: Learning what to share. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3098–3112. Association for Computational Linguistic… find
  2. Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Transactions of the Association for Computational Linguistics, 6:557–570. find our card
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  5. Maud Ehrmann, Marco Turchi, and Ralf Steinberger. 2011. Building a multilingual named entity-annotated corpus using annotation projection. In Proceedings of the International Conference Recent Advances in Natural Language Processing 2011, pages 118–124. Association for Computational Linguistics. find
  6. Akiko Eriguchi, Melvin Johnson, Orhan Firat, Hideto Kazawa, and Wolfgang Macherey. 2018. Zero-shot cross-lingual classification using multilingual neural machine translation. CoRR, abs/1809.04686. — [arXiv:1809.04686](https://arxiv.org/abs/1809.04686) find
  7. Donghui Feng, Yajuan Lv, and Ming Zhou. 2004. A new approach for English-Chinese named entity alignment. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 372–379. Association for Computational Linguistics. find
  8. Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, and Yun-Nung Chen. 2018. Slot-gated modeling for joint slot filling and intent prediction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… find our card
  9. Xiaodong He, Li Deng, Dilek Hakkani-Tur, and Gokhan Tur. 2013. Multi-style adaptive training for robust cross-lingual spoken language understanding. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8342– 8346. IEEE. find
  10. Alankar Jain, Bhargavi Paranjape, and Zachary C. Lipton. 2019. Entity projection via machine translation for cross-lingual NER. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-I… find
  11. Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR). find
  12. Ying Lin, Shengqi Yang, Veselin Stoyanov, and Heng Ji. 2018. A multi-lingual multi-task architecture for low-resource sequence labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 799–809. Association for Computational Li… find
  13. Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. find
  14. Interspeech 2016, pages 685–689. find
  15. Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017. Cheap translation for cross-lingual named entity recognition. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2536–2545. Association for Computational Linguistics. find
  16. Jian Ni, Georgiana Dinu, and Radu Florian. 2017. Weakly supervised cross-lingual named entity recognition via effective annotation and representation projection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1470–1480. Assoc… find
  17. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics. find our card
  18. Barbara Plank and Željko Agić. 2018. Distant supervision from disparate sources for low-resource part-of-speech tagging. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 614–620. Association for Computational Linguistics. find
  19. P. J. Price. 1990. Evaluation of spoken language systems: the ATIS domain. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990. find
  20. Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. 2019. A stack-propagation framework with token-level intent detection for spoken language understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferenc… find our card
  21. Doaa Samy, Antonio Moreno, and Jose M Guirao. 2005. A proposal for an arabic named entity tagger leveraging a parallel corpus. In International Conference RANLP, Borovets, Bulgaria, pages 459–465. find
  22. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681. find
  23. Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long… find our card
  24. Rushin Shah, Bo Lin, Anatole Gershman, and Robert Frederking. 2010. Synergy: a named entity recognition system for resource-scarce languages such as swahili using online machine translation. In Proceedings of the Second Workshop on African Language Technology (AfLaT 2010), pages 21–26. find
  25. Karan Singla, Dogan Can, and Shrikanth Narayanan. 2018. A multi-task approach to learning multilingual representations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 214– 220. Association for Computational Linguistics. find
  26. Oscar Täckström, Dipanjan Das, Slav Petrov, Ryan Mc-Donald, and Joakim Nivre. 2013. Token and type constraints for cross-lingual part-of-speech tagging. Transactions of the Association for Computational Linguistics, 1:1–12. find
  27. Chen-Tse Tsai, Stephen Mayhew, and Dan Roth. 2016. Cross-lingual named entity recognition via wikification. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 219–228. Association for Computational Linguistics. find
  28. Gokhan Tur, Dilek Hakkani-Tür, and Larry Heck. 2010. What is left to be understood in ATIS? In 2010 IEEE Spoken Language Technology Workshop, pages 19– 24. IEEE. find
  29. Shyam Upadhyay, Manaal Faruqui, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2018. (almost) zero-shot cross-lingual spoken language understanding. In Proceedings of the IEEE ICASSP. find
  30. Xiaojun Wan. 2009. Co-training for cross-lingual sentiment classification. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 235–243. Association for Computational Linguistics. find our card
  31. Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A. Smith, and Jaime Carbonell. 2018. Neural cross-lingual named entity recognition with minimal resources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 369–379. Association for Computational Linguistics… find our card
  32. Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. In International Conference on Learning Representations. find
  33. David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of the First International Conference on Human Language Technology Research. find our card
  34. Katherine Yu, Haoran Li, and Barlas Oguz. 2018. Multilingual seq2seq training with similarity loss for cross-lingual document classification. In Proceedings of The Third Workshop on Representation Learning for NLP, pages 175–179. Association for Computational Linguistics. find
  35. Ayah Zirikly and Masato Hagiwara. 2015. Cross-lingual transfer of named entity recognizers without parallel corpora. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short P… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2020-end-end-slot-alignment-recognition/