Elamur Research
☰
home / papers / 2021 / MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark

MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.eacl-main.257.pdf

Summary

Semantic parsers for task-oriented assistants are well developed for English, but other languages have little annotated data. Earlier multilingual sets covered only a few languages, held small amounts of data per language, and used only flat intent-and-slot labels, with no nested queries. The authors address this with MTOP, a nearly parallel corpus of about 100k utterances in English, German, French, Spanish, Hindi and Thai. It spans 11 domains and includes both flat and compositional (decoupled, TOP-style) annotations.

The English data was crowdsourced and double-annotated. Professional translators then rendered the utterances and their slot spans into the other languages, the annotation trees were rebuilt from the English structure, and two quality-control rounds removed weak examples.

The authors benchmark several multilingual pre-trained models, including XLM-R, mBART, CRISS and MARGE, in three settings: in-language, multilingual and zero-shot. They also propose a distant-supervision method that reduces noise when slot labels are projected onto machine-translated data. They report new state-of-the-art results on Multilingual ATIS and Multilingual TOP. They also show that zero-shot transfer comes reasonably close to training on target-language data, which lowers the cost of supporting new languages.

Key points

  • MTOP dataset: 6 languages, 11 domains, 117 intents and 78 slots, split roughly 70/10/20 into train, eval and test. According to the authors, it is the first multilingual dataset with compositional nested-query annotations.
  • Data quality: In the second review round, 83% of the translated data passed the quality check, and low-quality items were dropped. Each language took 15–25 days and involved three translators.
  • Existing benchmarks: Models based on XLM-R raise Slot F1 by an average of +6.3 points over the best results previously published for Multilingual ATIS and Multilingual TOP.
  • Zero-shot transfer: Using automatic translation, alignment and distant supervision, the models reach 67.2% exact match averaged over 5 non-English languages. The best in-language model reaches 77.7%.
  • Model comparison: CRISS and MARGE lead in most languages, but XLM-R does best on Thai, the only language in this group that XLM-R saw during pre-training. Mismatches in Thai tokenization hurt the translation-based methods.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

123 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 118 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.eacl-main.257

@inproceedings{li2021mtop,
  title = {{MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark}},
  author = {Li, Haoran and Arora, Abhinav and Chen, Shuohui and Gupta, Anchit and Gupta, Sonal and Mehdad, Yashar},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.257},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.257}
}

References

37 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A Smith. 2016. Massively multilingual word embeddings. arXiv preprint arXiv:1602.01925. find our card
  2. Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics… find
  3. Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-desig… find our card
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  5. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 9-14, 2013, Westin Peachtree Plaza Hotel… find
  6. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In HLT-NAACL, pages 199–209. The Association for Computational Linguistics. find
  7. Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2787–2792. find
  8. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. arXiv preprint arXiv:2003.11080. find our card
  9. Giuseppe Castellucci, Valentina Bellomaria, Andrea Favalli, and Raniero Romagnoli. 2019. Multilingual intent detection and slot filling in a joint bert-based model. arXiv preprint arXiv:1907.02884. — [arXiv:1907.02884](https://arxiv.org/abs/1907.02884) find
  10. Pavel Izmailov, Dmitry Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence (UAI 2018). find our card
  11. Qian Chen, Zhu Zhuo, and Wen Wang. 2019. Bert for joint intent classification and slot filling. arXiv preprint arXiv:1902.10909. — [arXiv:1902.10909](https://arxiv.org/abs/1902.10909) find
  12. Diederik Kingma and Jimmy Ba. 2015. method for stochastic optimization. find
  13. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Adam: A Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems (NeurIPS). find
  14. Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In ICLR (Poster). OpenReview.net. find
  15. Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020a. Pre-training via paraphrasing. arXiv preprint arXiv:2006.15020. find
  16. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020b. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of th… find our card
  17. Zhouhan Lin, Minwei Feng, Cı́cero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open… find our card
  18. Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. In Interspeech 2016, 17th Annual Conference of the International Speech Communication Association, San Francisco, CA, USA, September 8-12, 2016, pages 685–689. ISCA. find
  19. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020a. Multilingual denoising pre-training for neural machine translation. CoRR, abs/2001.08210. find
  20. Zihan Liu, Genta Indra Winata, Zhaojiang Lin, Peng Xu, and Pascale Fung. 2020b. Attention-informed mixed-language training for zero-shot cross-lingual task-oriented dialogue systems. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications… find
  21. Grégoire Mesnil, Xiaodong He, Li Deng, and Yoshua Bengio. 2013. Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding. In INTERSPEECH, pages 3771–3775. find
  22. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. CoRR, abs/1309.4168. find
  23. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL. find
  24. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? In ACL (1), pages 4996–5001. Association for Computational Linguistics. find our card
  25. P. J. Price. 1990. Evaluation of spoken language systems: the atis domain. In HLT. Morgan Kaufmann. find
  26. Subendhu Rongali, Luca Soldaini, Emilio Monti, and Wael Hamza. 2020. Don’t parse, generate! a sequence to sequence architecture for task-oriented semantic parsing. arXiv preprint arXiv:2001.11458. — [arXiv:2001.11458](https://arxiv.org/abs/2001.11458) find
  27. Sebastian Ruder, Ivan Vulic, and Anders Sogaard. 2017. A survey of cross-lingual word embedding models. Cite arxiv:1706.04902. find
  28. Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019a. Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lon… find our card
  29. Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019b. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan… find
  30. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401. find
  31. Chau Tran, Yuqing Tang, Xian Li, and Jiatao Gu. 2020. Cross-lingual retrieval for iterative self-supervised training. arXiv preprint arXiv:2006.09526. — [arXiv:2006.09526](https://arxiv.org/abs/2006.09526) find
  32. Shyam Upadhyay, Manaal Faruqui, Gökhan Tür, Dilek Hakkani-Tür, and Larry P. Heck. 2018. (almost) zero-shot cross-lingual spoken language understanding. In ICASSP, pages 6034–6038. IEEE. find
  33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  34. Jason D. Williams, Antoine Raux, and Matthew Henderson. 2016. The dialog state tracking challenge series: A review. D&D, 7(3):4–33. find
  35. Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2019. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962. find
  36. Xiaodong Zhang and Houfeng Wang. 2016. A joint model of intent determination and slot filling for spoken language understanding. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 2993–2999. IJCAI/AAAI Pr… find
  37. Su Zhu and Kai Yu. 2017. Encoder-decoder with focus-mechanism for sequence labelling based spoken language understanding. In ICASSP, pages 5675–5679. IEEE. find