Elamur Research⌕
☰
home / papers / 2021 / Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions

Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions

Paper · 2021 · Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin et al.

Summary

Queries to open-domain dialogue systems are often too vague to be answered right away. Unlike in web search, a bot in a conversation can give only one answer, so a mistake costs more. The authors propose that in such cases the system should first clarify the user's intent with a clarifying question. The biggest obstacle for this line of research is the lack of suitable data. The authors therefore collected and released the ClariQ corpus. It contains queries labeled for whether they need clarification, sets of possible clarifying questions for different facets of a topic, and user answers. Besides single-turn dialogues, they also obtained multi-turn ones: crowdworkers conversed with the two best agents. The task is split into two steps: deciding whether clarification is needed, and selecting the most appropriate question from a question bank. Evaluation also has two stages. Offline, they automatically measure how a question affects document retrieval and how completely the system retrieves relevant questions. Online, quality is judged by humans via Acute-eval. Among the compared models, ELECTRA performed best, and the human judgments agreed with the automatic ones. The corpus became the basis of the ConvAI3 challenge.

Key points

  • ClariQ contains about 15K single-turn and about 1.5M multi-turn dialogues on roughly 300 topics. The corpus, baseline code and offline evaluation pipeline are available on GitHub (aliannejadi/ClariQ).
  • The architecture has a user request understanding module (URU). It decides whether to pass the query to clarifying question generation (CQG) or directly to answer generation (AG). The paper focuses on the first two modules.
  • For single-turn dialogues, retrieval of relevant questions was evaluated with Recall@30 on the test set. Results: ELECTRA — 0.8744, Roberta — 0.8568, BERT — 0.8409, BM25 — 0.7682. Combining BERT or Roberta with BM25 did not improve quality.
  • For multi-turn dialogues, the main metric is NDCG@3. ELECTRA scored 0.0553, Roberta — 0.0522; MRR was 0.1798 and 0.1669, respectively.
  • In the human evaluation via Acute-eval, 120 pairs of dialogues were compared. ELECTRA beat Roberta in 56–59% of judgments on all five criteria: HU, EG, IN, KL and the new Clarification (CL) criterion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations🏢MicrosoftRedmond · United States378 works · 161,630 citations🎓University of AmsterdamAmsterdam · Netherlands151 works · 174,506 citations🎓University of GlasgowGlasgow · United Kingdom14 works · 746 citations🤝AIRI - Artificial Intelligence Research InstituteMoscow · Russia6 works · 140 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksDialogue systems · 654 works

Cite

Cite as Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton and Mikhail Burtsev. 2021. Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.367 · arXiv 2109.05794

@inproceedings{aliannejadi2021building,
  title = {{Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions}},
  author = {Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton and Mikhail Burtsev},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.367},
  eprint = {2109.05794},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.367}
}

References

61 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977.
  2. Mohammad Aliannejadi, Leif Azzopardi, Hamed Zamani, Paul Thomas, and Nick Craswell. 2021. Analysing mixed initiatives and search strategies during conversational search. In International Conference on Information and Knowledge Management (CIKM). find
  3. Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail S. Burtsev. 2020. ConvAI3: Generating clarifying questions for open-domain dialogue systems (ClariQ). arXiv preprint arXiv:2009.11352. — [arXiv:2009.11352](https://arxiv.org/abs/2009.11352) find
  4. Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 475–484. find
  5. Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017. What do you mean exactly? analyzing clarification questions in CQA. In Conference on Conference Human Information Interaction and Retrieval (CHIIR), pages 345–348. find
  6. Mikhail Burtsev, Aleksandr Chuklin, Julia Kiseleva, and Alexey Borisov. 2017. Search-oriented conversational AI (SCAI). In ACM SIGIR International Conference on Theory of Information Retrieval (IC-TIR), pages 333–334. find
  7. Mikhail Burtsev and Varvara Logacheva. 2020. Con-We publicly release the collected datasets, the versational intelligence challenge: Accelerating recode for training baselines, and our evaluation pro-search with crowd science and open source. AI Mag-cedure in order to push forward the state-of-the-a… find
  8. Mikhail Burtsev, Varvara Logacheva, Valentin Malykh, Iulian Vlad Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W Black, Alexander Rudnicky, and Yoshua Bengio. 2018. The first conversational intelligence challenge. In The NIPS’17 Competition: Building Intelligent Systems, pages 25–46. find
  9. Zi Chai and Xiaojun Wan. 2020. Learning to ask more: Semi-autoregressive sequential question generation under dual-graph interaction. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 225–237. find
  10. Di Jiang, Kenneth Wai-Ting Leung, Lingxiao Yang, and Wilfred Ng. 2015. Query suggestion with diversification and personalization. Knowledge-Based Systems, 89:553–568. find
  11. Makoto P Kato and Katsumi Tanaka. 2016. To suggest, or not to suggest for queries with diverse intents: Optimizing search result presentation. In ACM International Conference on Web Search and Data Mining (WSDM), pages 133–142. find
  12. Hugh A Chipman, Edward I George, Robert E McCulloch, et al. 2010. Bart: Bayesian additive regression trees. The Annals of Applied Statistics, 4(1):266– 298. find
  13. Julia Kiseleva, Kyle Williams, Ahmed Hassan Awadallah, Aidan C Crook, Imed Zitouni, and Tasos Anastasakos. 2016a. Predicting user satisfaction with intelligent assistants. In ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 45–54. find
  14. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations (ICLR). find
  15. Julia Kiseleva, Kyle Williams, Jiepu Jiang, Ahmed Hassan Awadallah, Aidan C Crook, Imed Zitouni, and Tasos Anastasakos. 2016b. Understanding user satisfaction with intelligent assistants. In Conference on Human Information Interaction and Retrieval (CHIIR), pages 121–130. find
  16. Charles L. A. Clarke, Nick Craswell, and Ian Soboroff. 2009. Overview of the TREC 2009 web track. In Text REtrieval Conference (TREC). find
  17. Jeff Dalton, Aleksandr Chuklin, Julia Kiseleva, and Mikhail Burtsev. 2020. Proceedings of the 5th international workshop on search-oriented conversational AI (SCAI). In International Workshop on Search-Oriented Conversational AI (SCAI). find
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171–…
  19. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2020. The second conversational intelligence challenge (ConvAI2). In The NeurIPS’18 Competition, pages 187–208.
  20. Rafal Kocielnik, Saleema Amershi, and Paul N Bennett. 2019. Will you accept an imperfect AI? exploring designs for adjusting end-user expectations of AI systems. In CHI Conference on Human Factors in Computing Systems (CHI), pages 1–14. find
  21. Weize Kong and James Allan. 2014. Extending faceted search to the general web. In ACM International Conference on Conference on Information and Knowledge Management (CIKM), pages 839– 848. find
  22. Weize Kong and James Allan. 2016. Precision-oriented query facet extraction. In ACM International on Conference on Information and Knowledge Management (CIKM), pages 1433–1442. find
  23. Antonios Minas Krasakis, Mohammad Aliannejadi, Nikos Voskarides, and Evangelos Kanoulas. 2020. Analysing the effect of clarifying questions on document ranking in conversational search. In ACM SI-GIR International Conference on the Theory of Information Retrieval (ICTIR), pages 129–132. find
  24. Yifan Gao, Piji Li, Irwin King, and Michael R. Lyu. 2019. Interconnected question generation with coreference alignment and conversation flow modeling. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4853–4862. find
  25. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguisti…
  26. Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazaré, and Jason Weston. 2019. Learning from dialogue after deployment: Feed yourself, chatbot! In Annual Meeting of the Association for Computational Linguistics (ACL), pages 3667–3684. find
  27. Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc’Aurelio Ranzato, and Jason Weston. 2017. Dialogue learning with human-in-the-loop. In International Conference on Learning Representations (ICLR). find
  28. Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020. Challenges in building intelligent open-domain dialog systems. ACM Transactions on Information Systems (TOIS), 38(3):1–32. find
  29. Margaret Li, Jason Weston, and Stephen Roller. 2019a. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087. — [arXiv:1909.03087](https://arxiv.org/abs/1909.03087) find
  30. Xiujun Li, Zachary C Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. 2016. A user simulator for task-completion dialogues. arXiv preprint arXiv:1612.05688. find
  31. Ziming Li, Julia Kiseleva, and Maarten de Rijke. 2019b. Dialogue generation: From imitation learning to inverse reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAI), pages 6722–6729. find
  32. Ziming Li, Sungjin Lee, Baolin Peng, Jinchao Li, Julia Kiseleva, Maarten de Rijke, Shahin Shayandeh, and Jianfeng Gao. 2020. Guided dialogue policy learning without adversarial learning in the loop. In Findings of the Association for Computational Linguistics (EMNLP), pages 2308–2317. find
  33. Ziming Li, Dookun Park, Julia Kiseleva, Young-Bum Kim, and Sungjin Lee. 2021. DEUS: A data-driven approach to estimate user satisfaction in multi-turn dialogues. arXiv preprint arXiv:2103.01287. — [arXiv:2103.01287](https://arxiv.org/abs/2103.01287) find
  34. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  35. Tom Lotze, Stefan Klut, Mohammad Aliannejadi, and Evangelos Kanoulas. 2021. Ranking clarifying questions based on predicted user engagement. arXiv preprint arXiv:2103.06192. — [arXiv:2103.06192](https://arxiv.org/abs/2103.06192) find
  36. Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic turing test: Learning to evaluate dialogue responses. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1116–1126.
  37. Clifford Nass and Youngme Moon. 2000. Machines and mindlessness: Social responses to computers. Journal of social issues, 56(1):81–103. find
  38. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2241–2252.
  39. Wenjie Ou and Yue Lin. 2020. A clarifying question selection system from ntes along in ConvAI3 challenge. arXiv preprint arXiv:2010.14202. — [arXiv:2010.14202](https://arxiv.org/abs/2010.14202) find
  40. Dragomir R. Radev, Hong Qi, Harris Wu, and Weiguo Fan. 2002. Evaluating web-based question answering systems. In International Conference on Language Resources and Evaluation (LREC), pages 1153–1156. find
  41. Filip Radlinski and Susan Dumais. 2006. Improving personalized web search using result diversification. In ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 691–692. find
  42. Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2737–2746. find
  43. Ehud Reiter. 2018. A structured review of the validity of bleu. Computational Linguistics, 44(3):393–401.
  44. Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at TREC-3. Nist Special Publication Sp, 109:109.
  45. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), page…
  46. Corby Rosset, Chenyan Xiong, Xia Song, Daniel Campos, Nick Craswell, Saurabh Tiwary, and Paul Bennett. 2020. Leading conversational search by suggesting useful questions. In The Web Conference, pages 1160–1170. find
  47. Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019a. Do massively pretrained language models make better storytellers? In Conference on Computational Natural Language Learning (CoNLL), pages 843–861.
  48. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019b. What makes a good conversation? how controllable attributes affect human judgments. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1702– 17…
  49. Ivan Sekulic, Mohammad Aliannejadi, and Fabio Crestani. 2021. User engagement prediction for clarification in search. In European Conference on Information Retrieval (ECIR), pages 619–633. find
  50. Amir Soleimani, Christof Monz, and Marcel Worring. 2021. NLQuAD: A non-factoid long question answering data set. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 1245–1255. find
  51. Shang-Yu Su, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Yun-Nung Chen. 2018. Discriminative deep dyna-q: Robust planning for dialogue policy learning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3813–3823. find
  52. Ryuichi Takanobu, Hanlin Zhu, and Minlie Huang. 2019. Guided dialog policy learning: Reward estimation for multi-domain task-oriented dialog. In Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages… find
  53. Jan Trienes and Krisztian Balog. 2019. Identifying unclear questions in community question answering websites. In European Conference on Information Retrieval (ECIR), pages 276–289. find
  54. David Vallet and Pablo Castells. 2012. Personalized diversification of search results. In ACM SIGIR conference on Research and Development in Information Retrieval (SIGIR), pages 841–850. find
  55. EM Voorhees. 1999. Proceedings of the 8th text retrieval conference. TREC-8 Question Answering Track Report, pages 77–82. find
  56. Yansen Wang, Chenyi Liu, Minlie Huang, and Liqiang Nie. 2018. Learning to ask questions in open-domain conversational systems with typed decoders. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2193–2203. find
  57. Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tie-Yan Liu. 2013. A theoretical analysis of NDCG type ranking measures. In Conference on Learning Theory (COLT), pages 25–54. find
  58. Hamed Zamani, Susan Dumais, Nick Craswell, Paul Bennett, and Gord Lueck. 2020a. Generating clarifying questions for information retrieval. In The Web Conference, pages 418–428. find
  59. Hamed Zamani, Gord Lueck, Everest Chen, Rodolfo Quispe, Flint Luu, and Nick Craswell. 2020b. MIM-ICS: A large-scale data collection for search clarification. In ACM International Conference on Information and Knowledge Management (CIKM), pages 3189–3196. find
  60. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2204– 2213.
  61. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT : Large-scale generative pre-training for conversational response generation. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 270…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aliannejadi-2021-building-evaluating-open-domain-dialogue/