Elamur Research⌕
☰
home / papers / 2023 / Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation

Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation

Paper · 2023 · Olga Majewska, Evgeniia Razumovskaia, Edoardo M. Ponti et al.

Summary

Multilingual task-oriented dialogue (ToD) systems lack good training and evaluation data. Datasets built from scratch tend to be small and cover few dialogue flows. Datasets translated from English often read unnaturally, miss local cultural context, and make cross-lingual scores look better than they really are. The authors propose a hybrid annotation pipeline to address both problems. They sample abstract dialogue schemata from the English Schema-Guided Dialogue dataset (SGD) and convert them automatically, using rewrite rules, into English outlines that state each turn's intent and slots. Native speakers then write the dialogues directly in their own language, adapting slot values such as cities to local geography and culture. The resulting resource, COD, supports intent detection, slot labeling, dialogue state tracking and end-to-end evaluation. A Russian pilot compared the pipeline with manual translation. Outline-based dialogues were annotated at a similar speed but were judged more natural and culturally familiar, with no loss in fluency. Benchmarks on COD show that translation-based test sets overestimate transfer performance, which makes COD a more realistic evaluation resource.

Key points

  • Coverage: COD spans Arabic, Indonesian, Russian and Kiswahili, plus the English source, across 11 domains. The languages come from four families and use Cyrillic, Arabic and Latin scripts.
  • Cost and preference: Annotation cost about 800 GBP per language. In side-by-side comparisons, raters preferred the outline-based dialogues over translated ones in more than 80% of cases.
  • Overestimation explained: Train and test sentence distributions differ much more in COD than in the translation-based data. This divergence explains why scores on translated test sets are inflated.
  • Baseline findings: The baselines were XLM-R for NLU and mT5 for end-to-end modeling, among others.
    • Zero-shot transfer beats translate-test for slot labeling, but the reverse holds for intent detection.
    • Reported gaps of roughly 10 points (intent detection) and 15 points (slot labeling) are widest for Kiswahili.
    • Cross-domain results fall well below in-domain results.
  • Availability: COD is also usable for cross-domain multilingual evaluation. Data and code are available at github.com/cambridgeltl/COD.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom381 works · 80,747 citations🎓University of CambridgeCambridge · United Kingdom256 works · 123,373 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 worksDialogue systems · 654 works

Cite

Cite as Olga Majewska, Evgeniia Razumovskaia, Edoardo M. Ponti, Ivan Vulić and Anna Korhonen. 2023. Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00539

@article{majewska2023cross,
  title = {{Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation}},
  author = {Olga Majewska, Evgeniia Razumovskaia, Edoardo M. Ponti, Ivan Vulić and Anna Korhonen},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00539},
  url = {https://doi.org/10.1162/tacl_a_00539}
}

References

69 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Duygu Altinok. 2018. An ontology-based dialogue management system for banking and finance dialogue systems. In Proceedings of the First Financial Narrative Processing Workshop (FNP), pages 1–9. find
  2. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020. Translation artifacts in cross-lingual transfer learning. In Proceedings of EMNLP 2020, pages 7674–7684. https://doi.org/10.18653/v1/2020.emnlp-main.618 ↗
  3. Leonard H. Babby and Richard D. Brecht. 1975. The syntax of voice in Russian. Language, pages 342–367. https://doi.org/10.2307/412860 — [doi:10.2307/412860](https://doi.org/10.2307/412860) ↗
  4. Valentina Bellomaria, Giuseppe Castellucci, Andrea Favalli, and Raniero Romagnoli. 2019. find
  5. Almawave-SLU: A new dataset for SLU in Italian. In Proceedings of the Sixth Italian Conference on Computational Linguistics. find
  6. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. TACL, 8:454–470. https://doi.org/10.1162/tacl a 00317 ↗
  7. Dan Bohus and Alexander I. Rudnicky. 2009. The RavenClaw dialog management framework: Architecture and systems. Computer Speech & Language, 23(3):332–361. https://doi.org/10.1016/j.csl.2008.10.001 — [doi:10.1016/j.csl.2008.10.001](https://doi.org/10.1016/j.csl.2008.10.001) ↗
  8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL 2020, pages 8440–8451. https://doi.org /… ↗
  9. Helene Bonneau-Maynard, Sophie Rosset, Christelle Ayache, A. Kuhn, and Djamel Mostefa. 2005. Semantic annotation of the French Media Dialog Corpus. In Proceedings of the Eurospeech 2005, pages 3457–3460. https://doi.org/10.21437/Interspeech .2005-312 — [doi:10.21437/interspeech](https://doi.org/10.21437/interspeech) ↗
  10. Mai Hoang Dao, Thinh Hung Truong, and Dat Quoc Nguyen. 2021. Intent detection and slot filling for Vietnamese. In Proceedings of Interspeech 2021, pages 4698–4702. find
  11. Paweł Budzianowski and Ivan Vulić. 2019. Hello, It’s GPT-2-How can I help you? Towards the use of pretrained language models for task-oriented dialogue systems. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 15–22. https://doi.org/10.18653/v1/D19-5602 ↗
  12. Kerstin Denecke, Mauro Tschanz, Tim Lucas Dorner, and Richard May. 2019. Intelligent conversational agents in healthcare: Hype or hope? Studies in Health Technology and Informatics, 259:77–84. find
  13. Jie Cao and Yi Zhang. 2021. A comparative study on schema-guided dialogue state tracking. In Proceedings of NAACL-HLT 2021, pages 782–796. https://doi.org/10.18653/v1/2021.naacl-main.62 ↗
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019, volume 1, pages 4171–4186.
  15. Giuseppe Castellucci, Valentina Bellomaria, Andrea Favalli, and Raniero Romagnoli. 2019. Multi-lingual intent detection and slot filling in a joint BERT-based model. arXiv preprint arXiv:1907.02884. — [arXiv:1907.02884](https://arxiv.org/abs/1907.02884) find
  16. Bosheng Ding, Junjie Hu, Lidong Bing, Mahani Aljunied, Shafiq Joty, Luo Si, and Chunyan Miao. 2022. GlobalWoZ: Globalizing MultiWoZ to develop multilingual task-oriented dialogue systems. In Proceedings of ACL 2022, pages 1639–1657. https://doi.org/10 .18653/v1/2022.acl-long.115 find
  17. Guan-Lin Chao and Ian Lane. 2019. BERT-DST: Scalable end-to-end dialogue state tracking with bidirectional encoder representations from transformer. In Proceedings of Interspeech 2019, pages 1468–1472. https://doi.org/10.21437/Interspeech.2019-1355 — [doi:10.21437/interspeech.2019-1355](https://doi.org/10.21437/interspeech.2019-1355) ↗
  18. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM Model 2. In Proceedings of NAACL-HLT 2013, pages 644–648. find
  19. Ana Paula Chaves, Eck Doerry, Jesse Egbert, and Marco Gerosa. 2019. It’s how you say it: Identifying appropriate register for chatbot language design. In Proceedings of HAI’19, HAI ’19, pages 102–109. https://doi.org/10.1145/3349537.3351901 — [doi:10.1145/3349537.3351901](https://doi.org/10.1145/3349537.3351901) ↗
  20. M. Amin Farajian, António V. Lopes, André F. T. Martins, Sameen Maruf, and Gholamreza Haffari. 2020. Findings of the WMT 2020 shared task on chat translation. In Proceedings of the Fifth Conference on Machine Translation, pages 65–75.
  21. Ana Paula Chaves and Marco Aurelio Gerosa. 2021. Why should we care about register? Reflections on chatbot language design. arXiv preprint arXiv:2104.14699. — [arXiv:2104.14699](https://arxiv.org/abs/2104.14699) find
  22. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. find
  23. Language-agnostic BERT sentence embedding. In Proceedings of ACL 2022, pages 878–891.
  24. Xiaodong He, Li Deng, Dilek Hakkani-Tur, and Gokhan Tur. 2013. Multi-style adaptive training for robust cross-lingual spoken language understanding. In Proceedings of ICASSP 2013, pages 8342–8346. IEEE. find
  25. Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, et al. 2022. MAS-SIVE: A 1M-Example multilingual natural language understanding dataset with 51 typologically-diverse languages. arXiv preprint arXiv:22… find
  26. Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24–27, 1990, pages 96–101. https://doi.org/10.3115/116580.116613 ↗
  27. Yu Gong, Xusheng Luo, Yu Zhu, Wenwu Ou, Zhao Li, Muhua Zhu, Kenny Q. Zhu, Lu Duan, and Xi Chen. 2019. Deep cascade multi-task learning for slot filling in Chinese E-commerce Shopping Guide Assistant. In Proceedings of AAAI 2019, volume 33, pages 6465–6472. https://doi.org/10 .1609/aaai.v33i01.330164… find
  28. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In Proceedings of ICLR 2020, pages 1–16.
  29. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of ICML 2020, pages 4411–4421. find
  30. Rob van der Goot, Ibrahim Sharaf, Aizhan Imankulova, Ahmet Üstün, Marija Stepanović, Alan Ramponi, Siti Oryza Khairunnisa, Mamoru Komachi, and Barbara Plank. 2021. From masked language modeling to translation: Non-English auxiliary tasks improve zero-shot spoken language understanding. In Proceed… find
  31. Chia-Chien Hung, Anne Lauscher, Ivan Vulić, Simone Ponzetto, and Goran Glavaš. 2022. Multi2WOZ: A robust multilingual dataset and conversational pretraining for task-oriented dialog. In Proceedings of NAACL-HLT 2022, pages 3687–3703. find
  32. Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and translationese in machine translation evaluation. In Proceedings of EMNLP 2020, pages 72–81. https://doi.org/10.18653/v1/2020 .emnlp-main.6 ↗
  33. Narendra Gupta, Gokhan Tur, Dilek Hakkani-Tur, Srinivas Bangalore, Giuseppe Riccardi, and Mazin Gilbert. 2005. The AT&T spoken language understanding system. IEEE Transactions on Audio, Speech, and Language Processing, 14(1):213–222. https://doi.org/10.1109/TSA.2005.854085 — [doi:10.1109/tsa.2005.854085](https://doi.org/10.1109/tsa.2005.854085) ↗
  34. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, Syste…
  35. John F. Kelley. 1984. An Iterative Design Methodology for User-friendly Natural Language Office Information Applications. ACM Transactions on Information Systems (TOIS), 2(1):26–41. https://doi.org/10.1145/357417.357420 ↗
  36. Dilek Hakkani-Tür, Gökhan Tür, Asli Celikyilmaz, Yun-Nung Chen, Jianfeng Gao, Li Deng, and Ye-Yi Wang. 2016. Multi-domain joint semantic frame parsing using bi-directional RNN-LSTM. In Proceedings of Interspeech 2016, pages 715–719. https://doi.org/10.21437/Interspeech.2016-402 — [doi:10.21437/interspeech.2016-402](https://doi.org/10.21437/interspeech.2016-402) ↗
  37. Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the ACL-HLT 2011, pages 1318–1326.
  38. Jitin Krishnan, Antonios Anastasopoulos, Hemant Purohit, and Huzefa Rangwala. 2021. Multilingual code-switching for zero-shot cross-lingual intent prediction and slot filling. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 211–223. https://doi.org/10.18653 /v1/2021… find
  39. Zihan Liu, Genta Indra Winata, Zhaojiang Lin, Peng Xu, and Pascale Fung. 2020b. Attention-informed mixed-language training for zero-shot cross-lingual task-oriented dialogue systems. In Proceedings of AAAI 2020, pages 8433–8440. https://doi.org/10 .1609/aaai.v34i05.6362 find
  40. Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of EMNLP-IJCNLP 20… find
  41. Laurens Van der Maaten and Geoffrey Hinton. 2012. Visualizing non-metric similarities in multiple maps. Machine Learning, 87(1):33–55. https://doi.org/10.1007/s10994 -011-5273-4 — [doi:10.1007/s10994](https://doi.org/10.1007/s10994) ↗
  42. Christian Muise, Tathagata Chakraborti, Shubham Agarwal, Ondrej Bajgar, Arunima Chaudhary, Luis A. Lastras-Montano, Josef Ondrej, Miroslav Vodolan, and Charlie Wiecha. 2019. Planning for goal-oriented dialogue systems. arXiv preprint arXiv:1910.08137. — [arXiv:1910.08137](https://arxiv.org/abs/1910.08137) find
  43. Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012. Language models for machine translation: Original vs. translated texts. Computational Linguistics, 38(4):799–825. https://doi.org/10.1162/COLI a 00111 ↗
  44. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of ACL 2002, pages 311–318. https://doi.org/10 .3115/1073083.1073135
  45. Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of EACL 2021, pages 2950–2962.
  46. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of EMNLP 2020, pages 2362–2376.
  47. Zhaojiang Lin, Andrea Madotto, Genta Winata, Peng Xu, Feijun Jiang, Yuxiang Hu, Chen Shi, and Pascale N. Fung. 2021. BiToD: A bilingual multi-domain dataset for task-oriented dialogue modeling. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. find
  48. Edoardo Maria Ponti, Julia Kreutzer, Ivan Vulić, and Siva Reddy. 2021. Modeling latent translations for cross-lingual transfer. arXiv preprint arXiv:2107.11353. — [arXiv:2107.11353](https://arxiv.org/abs/2107.11353) find
  49. Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually grounded reasoning across languages and cultures. In Proceedings of EMNLP 2021, pages 10467–10485. find
  50. Jun Quan, Shian Zhang, Qian Cao, Zizhong Li, and Deyi Xiong. 2020. RiSAWOZ: A large-scale multi-domain wizard-of-oz dataset with rich semantic annotations for task-oriented dialogue modeling. In Proceedings of EMNLP 2020, pages 930–940. https://doi.org/10.18653/v1/2020.emnlp-main.67 ↗
  51. Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019. Benchmarking natural language understanding services for building conversational agents. In Proceedings of IWSDS 2019, pages 165–183. find
  52. Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of AAAI 2020, Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis,…
  53. Evgeniia Razumovskaia, Goran Glavaš, Olga Majewska, Edoardo M. Ponti, Anna Korhonen, and Ivan Vulic. 2022a. Crossing the conversational chasm: A primer on natural language processing for multilingual task-oriented dialogue systems. Journal of Artificial Intelligence Research, 74:1351–1402. https://… find
  54. Shyam Upadhyay, Manaal Faruqui, Gokhan Tür, Hakkani-Tür Dilek, and Larry Heck. 2018. (Almost) zero-shot cross-lingual spoken language understanding. In Proceedings of ICASSP 2018, pages 6034–6038. find
  55. Vered Volansky, Noam Ordan, and Shuly Wintner. 2015. On the features of translationese. Digital Scholarship in the Humanities, 30(1):98–118. https://doi.org/10 .1093/llc/fqt031 find
  56. Evgeniia Razumovskaia, Ivan Vulić, and Anna Korhonen. 2022b. Data augmentation and learned layer aggregation for improved multilingual language understanding in dialogue. In Findings of ACL 2022, pages 2017–2033. https://doi.org/10 .18653/v1/2022.findings-acl.160 find
  57. Weijia Xu, Batool Haider, and Saab Mansour. 2020. End-to-end slot alignment and recognition for cross-lingual NLU. In Proceedings of EMNLP 2020, pages 5052–5063.
  58. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of NAACL-HLT 2021, pages 483–498.
  59. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-networks. In Proceedings of EMNLP-IJCNLP 2019, pages 3982–3992. https://doi.org/10.18653/v1/D19 -1410 ↗
  60. Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020. Multilingual universal sentence encoder for semantic retrieval. In Proceedings of ACL 2020, System Demonstrations, pages 87–9…
  61. Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings of NAACL-HLT 2019, pages 3795–3805. https://doi.org/10 .18653/v1/N19-1380
  62. Pararth Shah, Dilek Hakkani-Tür, Bing Liu, and Gökhan Tür. 2018. Bootstrapping a neural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning. In Proceedings of NAACL-HLT 2018: Industry Papers, pages 41–51. https://doi.org/10.18653/v1/N18-3006 ↗
  63. Steve Young, Milica Gašić, Blaise Thomson, and Jason D. Williams. 2013. POMDP-based statistical spoken dialog systems: A review. In Proceedings of the IEEE, 101(5):1160–1179. https://doi.org/10 .1109/JPROC.2012.2225812 find
  64. Aditya Siddhant, Melvin Johnson, Henry Tsai, Naveen Ari, Jason Riesa, Ankur Bapna, Orhan Firat, and Karthik Raman. 2020. Evaluating the cross-lingual effectiveness of massively multilingual neural machine translation. In Proceedings of AAAI 2020, pages 8854–8861. https://doi.org/10 .1609/aaai.v34i05…
  65. Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages…
  66. Raymond Hendy Susanto and Wei Lu. 2017. Neural architectures for multilingual seman-Wei-Nan Zhang, Zhigang Chen, Wanxiang Che, Guoping Hu, and Ting Liu. 2017. The first evaluation of Chinese human-computer dialogue technology. arXiv preprint arXiv:1709 .10217. find
  67. Qi Zhu, Kaili Huang, Zheng Zhang, Xiaoyan Zhu, and Minlie Huang. 2020. CrossWOZ: A large-scale Chinese cross-domain task-oriented dialogue dataset. TACL, 8:281–295. https://doi.org/10.1162/tacl a 00314 ↗
  68. Zhichang Zhang, Zhenwen Zhang, Haoyuan Chen, and Zhiman Zhang. 2019. A joint learning framework with BERT for spoken language understanding. IEEE Access, 7:168849–168858. https://doi.org/10 .1109/ACCESS.2019.2954766 find
  69. Lei Zuo, Kun Qian, Bowen Yang, and Zhou Yu. 2021. AllWOZ: Towards multilingual task-oriented dialog systems for all. arXiv preprint arXiv:2112.08333. — [arXiv:2112.08333](https://arxiv.org/abs/2112.08333) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/majewska-2023-cross-lingual-dialogue-dataset-creation/