Elamur Research⌕
☰
home / papers / 2022 / Are Current Task-oriented Dialogue Systems Able to Satisfy Impolite Users?

Are Current Task-oriented Dialogue Systems Able to Satisfy Impolite Users?

Paper · 2022 · Zhiqiang Hu, Roy Kaa-Wei Lee and Nancy F. Chen

Summary

Task-oriented dialogue (TOD) systems, which handle jobs like booking tickets or answering service questions, are usually built and tested on users who speak courteously. Real customers can be rushed, irritated, or sarcastic. This paper asks whether current TOD models still work when the user is rude. The authors started from MultiWOZ 2.2. Native English speakers rewrote sampled user turns so they sounded impolite but kept the same meaning and stayed fluent. Role-play scenarios helped them vary the tone. The result is presented as the first impolite task-oriented dialogue corpus. The authors used it to benchmark six state-of-the-art TOD systems and found that all of them struggle once the user's tone turns hostile. They also tried a cheap remedy: generating impolite training utterances with text style transfer. This improves results, but the problem remains far from solved. The released corpus and benchmark give the field a way to test how robust systems are to the manner of a request, not only its content.

Key points

  • Motivation from data. About 99% of MultiWOZ 2.2 user utterances were labelled polite by the Stanford Politeness classifier, a model learned from request data collected on Wikipedia. Existing benchmarks therefore contain almost no rudeness.
  • Corpus construction. Eight native English speakers produced over 10K impolite rewrites of user turns.
    • Each annotator saw the full dialogue history while rewriting.
    • Each rewrite had to meet three goals: be impolite, stay semantically close to the original, and be fluent.
    • Six role-play scenarios guided the rewrites, for example an impatient customer or a sarcastic person in a bad mood.
  • Quality control. Impoliteness is subjective, so annotators rated each other's rewrites at repeated checkpoints. Politeness was one of the criteria and used a 5-point scale.
  • Benchmark result. Six state-of-the-art TOD systems were evaluated on the corpus and all had difficulty with impolite requests. Figure 1 shows PPTOD answering a neutral request correctly but giving an irrelevant reply to the rude version.
  • Augmentation and tooling. The proposed augmentation uses text style transfer to create impolite user utterances for training. The custom annotation tool is to be released publicly.

Interesting ideas

  • I1. Rewrite an existing labelled benchmark in a new style while keeping its content, so every gold label stays valid and any drop in accuracy can be blamed on style alone; this gives a clean, paired robustness test.
  • I2. Hand annotators (or a generator) a set of distinct persona scenarios instead of one generic instruction, which pushes the rewrites toward more varied wording than a single prompt would produce.
  • I3. Before building a resource, run an existing classifier over the current data to measure how rare the target phenomenon is; this turns a hunch about a gap into a number.
  • I4. Separate what a request asks for from the tone it is said in: a rude and a polite version should yield the same task outcome.
  • I5. For subjective labels, have annotators rate each other's output at repeated checkpoints instead of trusting a single pass; this catches drift in how people interpret a fuzzy notion.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Rephrase school-test questions and fable QA items in rude, sarcastic or impatient wording and check that the reader still gives the same answer, as a tone-invariance negative control.
  • I2 (learn): Give the LLM teacher tiers several personas when generating silver paraphrases, so the training and evaluation data cover more registers than tidy textbook English.
  • I3 (frames): Profile the treebanks and test sets Mova uses for the share of hostile, ironic or indirect speech acts before claiming that crate prag handles them.
  • I4 (frames): Make the speech-act frame record tone or face-threat as its own attribute, keeping the core request frame (who wants what) identical across polite and impolite paired variants.
  • I5 (learn): Apply cross-rating between witness parsers or teacher tiers at fixed checkpoints when producing silver labels for subjective categories like irony or politeness.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 624 works

Cite

Cite as Zhiqiang Hu, Roy Kaa-Wei Lee and Nancy F. Chen. 2022. Are Current Task-oriented Dialogue Systems Able to Satisfy Impolite Users?. arXiv:2210.12942.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2210.12942 · arXiv 2210.12942

@misc{hu2022are,
  title = {{Are Current Task-oriented Dialogue Systems Able to Satisfy Impolite Users?}},
  author = {Zhiqiang Hu, Roy Kaa-Wei Lee and Nancy F. Chen},
  year = {2022},
  doi = {10.48550/arxiv.2210.12942},
  eprint = {2210.12942},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2210.12942}
}

References

56 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. H. Elder, A. O’Connor, and J. Foster, “How to make neural natural language generation as reliable as templates in task-oriented dialogue,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 2877–2888. find
  2. A. Balakrishnan, J. Rao, K. Upasani, M. White, and R. Subba, “Constrained decoding for neural nlg from compositional representations in task-oriented dialogue,” arXiv preprint arXiv:1906.07220, 2019. ↗
  3. Y. Li, K. Yao, L. Qin, W. Che, X. Li, and T. Liu, “Slot-consistent nlg for task-oriented dialogue systems with iterative rectification network,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 97–106. find
  4. S. Golovanov, R. Kurbanov, S. Nikolenko, K. Truskovskyi, A. Tselousov, and T. Wolf, “Large-scale transfer learning for natural language generation,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 6053–6058. find
  5. Y. Yang, Y. Li, and X. Quan, “Ubar: Towards fully end-to-end task-oriented dialog systems with gpt-2,” in AAAI, 2021. ↗
  6. E. Hosseini-Asl, B. McCann, C.-S. Wu, S. Yavuz, and R. Socher, “A simple language model for task-oriented dialogue,” Advances in Neural Information Processing Systems, vol. 33, pp. 20 179–20 191, 2020. ↗
  7. Z. Lin, A. Madotto, G. I. Winata, and P. Fung, “MinTL: Minimalist transfer learning for task-oriented dialogue systems,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics, Nov. 2020, pp. 3391–3405. [Online]. Available: https://aclanthology.org/2020.emnlp-main.273 ↗
  8. S. Gupta, M. A. Walker, and D. M. Romano, “How rude are you?: Evaluating politeness and affect in interaction,” in International Conference on Affective Computing and Intelligent Interaction. Springer, 2007, pp. 203–217. find
  9. C. Bothe, F. Garcia, A. Cruz Maya, A. K. Pandey, and S. Wermter, “Towards dialogue-based navigation with multivariate adaptation driven by intention and politeness for social robots,” in International Conference on Social Robotics. Springer, 2018, pp. 230–240. find
  10. K. Mishra, M. Firdaus, and A. Ekbal, “Please be polite: Towards building a politeness adaptive dialogue system for goal-oriented conversations,” Neurocomputing, vol. 494, pp. 242–254, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231222003952 ↗
  11. Y. Su, L. Shu, E. Mansimov, A. Gupta, D. Cai, Y.-A. Lai, and Y. Zhang, “Multi-task pre-training for plug-and-play task-oriented dialogue system,” arXiv preprint arXiv:2109.14739, 2021. ↗
  12. P. Budzianowski, T.-H. Wen, B.-H. Tseng, I. Casanueva, S. Ultes, O. Ramadan, and M. Gašić, “MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 5016–5026. [Online]. Available: https://aclanthology.org/D18-1547 ↗
  13. X. Zang, A. Rastogi, S. Sunkara, R. Gupta, J. Zhang, and J. Chen, “MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines,” in Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI. Online: Association for Computational Linguistics, Jul. 2020, pp. 109– 117. [Online]. Available: https://aclanthology.org/2020.nlp4convai-1.13 ↗
  14. A. Rastogi, X. Zang, S. Sunkara, R. Gupta, and P. Khaitan, “Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, 2020, pp. 8689–8696. ↗
  15. Q. Zhu, K. Huang, Z. Zhang, X. Zhu, and M. Huang, “Crosswoz: A large-scale chinese cross-domain task-oriented dialogue dataset,” Transactions of the Association for Computational Linguistics, vol. 8, pp. 281–295, 2020. ↗
  16. Y. Lee, “Improving end-to-end task-oriented dialog system with a simple auxiliary task,” in Findings of the Association for Computational Linguistics: EMNLP 2021. Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 1296–1303. [Online]. Available: https://aclanthology.org/2021.findings-emnlp.112 ↗
  17. Y. Zhang, Z. Ou, and Z. Yu, “Task-oriented dialog systems that consider multiple appropriate responses under the same context,” in AAAI, 2020. find
  18. Y. Zhang, Z. Ou, M. Hu, and J. Feng, “A probabilistic end-to-end task-oriented dialog model with latent belief states towards semi-supervised learning,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics, Nov. 2020, pp. 9207–9219. [Online]. Available: https://aclanthology.org/2020.emnlp-main.740 ↗
  19. A. Bordes, Y.-L. Boureau, and J. Weston, “Learning end-to-end goal-oriented dialog,” arXiv preprint arXiv:1605.07683, 2016. ↗
  20. W. Lei, X. Jin, M.-Y. Kan, Z. Ren, X. He, and D. Yin, “Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 1437–1447. [Online]. Available: https://aclanthology.org/P18-1133 ↗
  21. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. find
  22. J. Kulhánek, V. Hudeček, T. Nekvinda, and O. Dušek, “AuGPT: Auxiliary tasks and data augmentation for end-to-end dialogue with pre-trained language models,” in Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI. Online: Association for Computational Linguistics, Nov. 2021, pp. 198–210. [Online]. Available: https://aclanthology.org/2021.nlp4convai-1.19 ↗
  23. S. Edunov, M. Ott, M. Auli, and D. Grangier, “Understanding back-translation at scale,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 489–500. [Online]. Available: https://aclanthology.org/D18-1045 ↗
  24. W. Zhu, K. Mo, Y. Zhang, Z. Zhu, X. Peng, and Q. Yang, “Flexible end-to-end dialogue system for knowledge grounded conversation,” arXiv preprint arXiv:1709.04264, 2017. ↗
  25. M. Eric and C. D. Manning, “Key-value retrieval networks for task-oriented dialogue,” arXiv preprint arXiv:1705.05414, 2017. ↗
  26. M. Ghazvininejad, C. Brockett, M.-W. Chang, B. Dolan, J. Gao, W.-t. Yih, and M. Galley, “A knowledge-grounded neural conversation model,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018. ↗
  27. K. Hua, Z. Feng, C. Tao, R. Yan, and L. Zhang, “Learning to detect relevant contexts and knowledge for response selection in retrieval-based dialogue systems,” in Proceedings of the 29th ACM international conference on information & knowledge management, 2020, pp. 525– 534. find
  28. A. Madotto, S. Cahyawijaya, G. I. Winata, Y. Xu, Z. Liu, Z. Lin, and P. Fung, “Learning knowledge bases with parameters for task-oriented dialogue systems,” in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 2372–2394. find
  29. T. Zhao and M. Eskenazi, “Towards end-to-end learning for dialog state tracking and management using deep reinforcement learning,” in Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2016, pp. 1–10. find
  30. X. Li, Z. C. Lipton, B. Dhingra, L. Li, J. Gao, and Y.-N. Chen, “A user simulator for task-completion dialogues,” arXiv preprint arXiv:1612.05688, 2016. ↗
  31. B. Liu and I. Lane, “Iterative policy learning in end-to-end trainable task-oriented neural dialog models,” in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 482–489. find
  32. Y. Ma, K. L. Nguyen, F. Z. Xing, and E. Cambria, “A survey on empathetic dialogue systems,” Information Fusion, vol. 64, pp. 50–70, 2020. ↗
  33. W. Shi and Z. Yu, “Sentiment adaptive end-to-end dialog systems,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 1509–1519. [Online]. Available: https://aclanthology.org/P18-1140 ↗
  34. S. Feng, N. Lubis, C. Geishauser, H.-c. Lin, M. Heck, C. van Niekerk, and M. Gašić, “Emowoz: A large-scale corpus and labelling scheme for emotion in task-oriented dialogue systems,” arXiv preprint arXiv:2109.04919, 2021. ↗
  35. D. Li, Y. Li, and S. Wang, “Interactive double states emotion cell model for textual dialogue emotion prediction,” Knowledge-Based Systems, vol. 189, p. 105084, 2020. ↗
  36. P. Brown and S. C. Levinson, “Universals in language usage: Politeness phenomena,” in Questions and politeness: Strategies in social interaction. Cambridge University Press, 1978, pp. 56–311. find
  37. C. Danescu-Niculescu-Mizil, M. Sudhof, D. Jurafsky, J. Leskovec, and C. Potts, “A computational approach to politeness with application to social factors,” in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Sofia, Bulgaria: Association for Computational Linguistics, Aug. 2013, pp. 250–259. [Online]. Available: https://aclanthology.org/P13-1025 ↗
  38. H. Golchha, M. Firdaus, A. Ekbal, and P. Bhattacharyya, “Courteously yours: Inducing courteous behavior in customer care responses using reinforced pointer generator network,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 851–860. find
  39. T. Niu and M. Bansal, “Polite dialogue generation without parallel data,” Transactions of the Association for Computational Linguistics, vol. 6, pp. 373–389, 2018. ↗
  40. A. Madaan, A. Setlur, T. Parekh, B. Póczos, G. Neubig, Y. Yang, R. Salakhutdinov, A. W. Black, and S. Prabhumoye, “Politeness transfer: A tag and generate approach,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 1869–1881.
  41. Y. Hou, Y. Liu, W. Che, and T. Liu, “Sequence-to-sequence data augmentation for dialogue language understanding,” in Proceedings of the 27th International Conference on Computational Linguistics. Santa Fe, New Mexico, USA: Association for Computational Linguistics, Aug. 2018, pp. 1234–1245. [Online]. Available: https://aclanthology.org/C18-1105 ↗
  42. M. Gritta, G. Lampouras, and I. Iacobacci, “Conversation graph: Data augmentation, training, and evaluation for non-deterministic dialogue management,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 36–52, 2021. [Online]. Available: https://aclanthology.org/2021.tacl-1.3 ↗
  43. G. Kurata, B. Xiang, and B. Zhou, “Labeled data generation with encoder-decoder lstm for semantic slot filling.” in INTERSPEECH, 2016, pp. 725–729. find
  44. Z. Hu, R. Lee, C. Aggarwal, and A. Zhang, “Text style transfer: a review and experimental evaluation (2020),” arXiv preprint arXiv:2010.12742. ↗
  45. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020. [Online]. Available: http://jmlr.org/papers/v21/20-074.html ↗
  46. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, Jul. 2020, pp. 7871–7880. [Online]. Available: https://aclanthology.org/2020.acl-main.703 ↗
  47. D. Li, Y. Zhang, Z. Gan, Y. Cheng, C. Brockett, B. Dolan, and M.-T. Sun, “Domain adaptive text style transfer,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3304–3313. [Online]. Available: https://aclanthology.org/D19-1325 ↗
  48. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018. ↗
  49. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics, Jul. 2002, pp. 311–318. [Online]. Available: https://aclanthology.org/P02-1040 ↗
  50. Z. Hu, R. K.-W. Lee, C. C. Aggarwal, and A. Zhang, “Text style transfer: A review and experimental evaluation,” ACM SIGKDD Explorations Newsletter, vol. 24, no. 1, pp. 14–45, 2022. find
  51. W. Sun, S. Zhang, K. Balog, Z. Ren, P. Ren, Z. Chen, and M. de Rijke, “Simulating user satisfaction for the evaluation of task-oriented dialogue systems,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021, pp. 2499–2506. find
  52. B. Peng, C. Li, Z. Zhang, C. Zhu, J. Li, and J. Gao, “Raddle: An evaluation benchmark and analysis platform for robust task-oriented dialog systems,” arXiv preprint arXiv:2012.14666, 2020. ↗
  53. J. Deriu, A. Rodrigo, A. Otegi, G. Echegoyen, S. Rosset, E. Agirre, and M. Cieliebak, “Survey on evaluation methods for dialogue systems,” Artificial Intelligence Review, vol. 54, no. 1, pp. 755–810, 2021. ↗
  54. T. Nekvinda and O. Dušek, “Shades of bleu, flavours of success: The case of multiwoz,” arXiv preprint arXiv:2106.05555, 2021. ↗
  55. S. Mehri, T. Srinivasan, and M. Eskenazi, “Structured fusion networks for dialog,” arXiv preprint arXiv:1907.10016, 2019. ↗
  56. N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. [Online]. Available: https://arxiv.org/abs/1908.10084 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hu-2022-current-task-oriented-dialogue-systems/