Elamur Research⌕
☰
home / papers / 2023 / A Survey on Asking Clarification Questions Datasets in Conversational Systems

A Survey on Asking Clarification Questions Datasets in Conversational Systems

Paper · 2023 · Hossein A. Rahmani, Xi Wang, Yue Feng et al.

Summary

Research on asking clarification questions (ACQs) in conversational systems is hard to compare across studies. Studies use different datasets, experimental setups and evaluation protocols. This survey reviews the field from the data side, drawing on 77 papers from strong venues. It concentrates on Conv. Search and Conv. QA, leaving aside task-oriented dialogue and social chatbots because they have few relevant resources. The authors catalogue publicly available ACQ datasets released since 2015 and compare their size, source, recency and semantic similarity. They map each dataset onto three subtasks:

  • deciding whether clarification is needed,
  • producing or selecting a clarification question,
  • predicting user satisfaction with that question.

They then run representative baselines on these datasets, including classical ML, language models and BM25 with Doc2Query expansion, and release the code. The results vary considerably from one dataset to another. Only two datasets support satisfaction prediction, and very few record real user–system interaction. The paper therefore argues for a shared benchmark, a common evaluation framework, large human-to-machine data and multi-modal ACQ datasets.

Key points

  • The survey covers 13 datasets.
    • Conv. Search: ClariT, Qulac, ClariQ, TavakoliCQ, MIMICS, MANtIS, ClariQ-FKw, MSDialog, MIMICS-Duo.
    • Conv. QA: ClarQ, RaoCQ, AmazonCQ, CLAQUA.
    • Datasets older than 2015 or not publicly available were excluded.
  • Datasets fall into three size tiers by number of clarification questions.
    • Large (>10K): ClariT, MIMICS, ClarQ, RaoCQ, AmazonCQ, CLAQUA. Every Conv. QA set is in this tier.
    • Medium: Qulac, ClariQ, TavakoliCQ, ClariQ-FKw, MIMICS-Duo.
    • Small: MANtIS, MSDialog.
  • The three subtasks are T1 (clarification need prediction), T2 (asking clarification questions, by generation or ranking) and T3 (user satisfaction with clarification questions).
  • Only MIMICS and MIMICS-Duo support T3, and the results on them diverge.
    • On MIMICS, distilBERT reaches F1 0.939 against 0.7758 for MultinomialNB.
    • On MIMICS-Duo, both models score low (F1 about 0.28 vs 0.23), and MultinomialNB has the higher precision.
  • Doc2Query expansion adds little to BM25 on datasets whose clarification questions resemble each other. It helps more on the more distinctive TavakoliCQ and MANtIS.

Interesting ideas

  • I1. Split clarification into three separately evaluated steps (is a question needed, which question, did it satisfy), so each step can fail and be measured independently
  • I2. Visualise the semantic encodings of several datasets together and use their closeness to predict where a method will help; here, expansion gained more on corpora that looked distinct
  • I3. Always run a cheap classical baseline next to the neural model on every dataset, because the stronger model does not win on every set
  • I4. Treat inconsistent dataset formats and per-paper custom metrics as a core obstacle, and fix them with one shared format and evaluation protocol

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): A reader facing an ambiguous test item could first decide explicitly whether the text underdetermines the answer, then name the missing slot, and log both decisions as explainable steps.
  • I2 (learn): Embedding treebanks and test sets on one map could predict where new silver data will pay off and warn when evaluation sets are near-duplicates of training data.
  • I3 (learn): Pair each transformer result with a simple perceptron or naive baseline as a sanity control, flagging datasets where the gain disappears.
  • I4 (tree): Converting all treebanks into one UD dialect and scoring them with a single fixed LAS script keeps parser comparisons across corpora honest.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University College LondonLondon · United Kingdom88 works · 19,432 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 624 works

Cite

Cite as Hossein A. Rahmani, Xi Wang, Yue Feng, Qiang Zhang, Emine Yilmaz and Aldo Lipani. 2023. A Survey on Asking Clarification Questions Datasets in Conversational Systems. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.152

@inproceedings{rahmani2023survey,
  title = {{A Survey on Asking Clarification Questions Datasets in Conversational Systems}},
  author = {Hossein A. Rahmani, Xi Wang, Yue Feng, Qiang Zhang, Emine Yilmaz and Aldo Lipani},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.152},
  url = {https://doi.org/10.18653/v1/2023.acl-long.152}
}

References

65 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, 4 https://www.amazon.science/alexa-prize/ taskbot-challenge Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. {TensorFlow}: a system for {Large-Scale} machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265–283. ↗
  2. Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2020. Convai3: Generating clarifying questions for open-domain dialogue systems (clariq). arXiv preprint arXiv:2009.11352. ↗
  3. Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2021. Building and evaluating open-domain dialogue corpora with clarifying questions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4473–4484. find
  4. Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), SIGIR ’19. find
  5. Giambattista Amati, Giuseppe Amodeo, Marco Bianchi, Carlo Gaibisso, and Giorgio Gambosi. 2008. Fub, iasi-cnr and university of tor vergata at trec 2008 blog track. Technical report, FONDAZIONE UGO BORDONI ROME (ITALY). find
  6. Gianni Amati and Cornelis Joost Van Rijsbergen. 2002. Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Transactions on Information Systems (TOIS), 20(4):357–389. ↗
  7. Avishek Anand, Lawrence Cavedon, Hideo Joho, Mark Sanderson, and Benno Stein. 2020. Conversational search (dagstuhl seminar 19461). In Dagstuhl Reports, volume 9. Schloss Dagstuhl-Leibniz-Zentrum für Informatik. ↗
  8. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  9. Steven M Beitzel. 2006. On understanding and classifying web queries. Illinois Institute of Technology. find
  10. Keping Bi, Qingyao Ai, and W Bruce Croft. 2021. Asking clarifying questions based on negative feedback in conversational search. In Proc. of ICTIR. find
  11. Leo Breiman. 2001. Random forests. Machine learning, 45(1):5–32. find
  12. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning, 20(3):273–297. ↗
  13. Yashar Deldjoo, Johanne R Trippas, and Hamed Zamani. 2021. Towards multi-modal conversational information seeking. In Proceedings of the 44th International ACM SIGIR conference on research and development in Information Retrieval, pages 1577– 1587. find
  14. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. ↗
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  16. Jiwei Li, Alexander H Miller, Sumit Chopra, Marc’Aurelio Ranzato, and Jason Weston. 2017. Dialogue learning with human-in-the-loop. In Proceedings of the 5th International Conference on Learning Representations, ICLR 2017. ↗
  17. Yue Feng, Hossein A Rahmani, Aldo Lipani, and Emine Yilmaz. 2023. Towards asking clarification questions for information seeking on task-oriented dialogues. arXiv preprint arXiv:2305.13690. ↗
  18. Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neural approaches to conversational ai. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 1371– 1374. ↗
  19. Margaret Li, Jason Weston, and Stephen Roller. 2019. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087. ↗
  20. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  21. Jianfeng Gao, Michel Galley, and Lihong Li. 2019. Neural approaches to conversational AI: Question answering, task-oriented dialogues and social chatbots. Now Foundations and Trends. ↗
  22. Wei-Yin Loh. 2011. Classification and regression trees. Wiley interdisciplinary reviews: data mining and knowledge discovery, 1(1):14–23. ↗
  23. Kalervo Jarvelin. 2000. Ir evaluation methods for retrieving highly relevant documents. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 2000. find
  24. Craig Macdonald and Nicola Tonellotto. 2020. Declarative experimentation ininformation retrieval using pyterrier. In Proceedings of ICTIR 2020. find
  25. Kalervo Järvelin and Jaana Kekäläinen. 2017. Ir evaluation methods for retrieving highly relevant documents. In ACM SIGIR Forum, volume 51, pages 243–250. ACM New York, NY, USA. ↗
  26. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39– 48. ↗
  27. Antonios Minas Krasakis, Mohammad Aliannejadi, Nikos Voskarides, and Evangelos Kanoulas. 2020. Analysing the effect of clarifying questions on document ranking in conversational search. In Proc. of ICTIR. find
  28. Vaibhav Kumar and Alan W Black. 2020. Clarq: A large-scale and diverse dataset for clarification question generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7296–7301. find
  29. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. ↗
  30. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  31. Christopher D Manning. 2008. Introduction to information retrieval. Syngress Publishing,. ↗
  32. Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pages 43–52. ↗
  33. Julian McAuley and Alex Yang. 2016. Addressing complex and subjective product-related queries with customer reviews. In Proceedings of the 25th International Conference on World Wide Web, pages 625–635. find
  34. Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction. arXiv preprint arXiv:1904.08375. ↗
  35. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. ↗
  36. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  37. Gustavo Penha, Alexandru Balan, and Claudia Hauff. 2019. Introducing mantis: a novel multi-domain information seeking dialogues dataset. arXiv preprint arXiv:1912.04639. ↗
  38. Chen Qu, Liu Yang, W Bruce Croft, Johanne R Trippas, Yongfeng Zhang, and Minghui Qiu. 2018. Analyzing and characterizing user intent in information-seeking conversations. In The 41st international acm sigir conference on research & development in information retrieval, pages 989–992. find
  39. Dragomir R Radev, Hong Qi, Harris Wu, and Weiguo Fan. 2002. Evaluating web-based question answering systems. In LREC. Citeseer. find
  40. Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2737–2746. ↗
  41. Sudha Rao and Hal Daumé III. 2019. Answer-based adversarial training for generating clarification questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 143–155. ↗
  42. Corbin Rosset, Chenyan Xiong, Xia Song, Daniel Campos, Nick Craswell, Saurabh Tiwary, and Paul Bennett. 2020. Leading conversational search by suggesting useful questions. In Proceedings of The Web Conference 2020, pages 1160–1170. find
  43. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. ↗
  44. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. arXiv preprint arXiv:1902.08654. ↗
  45. Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani. 2021. Towards facet-driven generation of clarifying questions for conversational search. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval, pages 167–175. find
  46. Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani. 2022. Exploiting document-based features for clarification in conversational search. In European Conference on Information Retrieval. find
  47. Taihua Shao, Fei Cai, Wanyu Chen, and Honghui Chen. 2022. Self-supervised clarification question generation for ambiguous multi-turn conversation. Information Sciences, 587:626–641. ↗
  48. Zhengxiang Shi, Yue Feng, and Aldo Lipani. 2022. Learning to execute or ask clarification questions. arXiv preprint arXiv:2204.08373. ↗
  49. Zhengxiang Shi, Jerome Ramos, To Eun Kim, Xi Wang, Hossein A Rahmani, and Aldo Lipani. 2023. When and what to ask through world states and text instructions: Iglu nlp challenge solution. arXiv preprint arXiv:2305.05754. ↗
  50. Leila Tavakoli. 2020. Generating clarifying questions in conversational search systems. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 3253–3256. find
  51. Leila Tavakoli, Johanne R Trippas, Hamed Zamani, Falk Scholer, and Mark Sanderson. 2022. Mimics-duo: Offline & online evaluation of search clarification. arXiv preprint arXiv:2206.04417. ↗
  52. Leila Tavakoli, Hamed Zamani, Falk Scholer, William Bruce Croft, and Mark Sanderson. 2021. Analyzing clarification in asynchronous information-seeking conversations. Journal of the Association for Information Science and Technology. ↗
  53. Johanne R Trippas, Damiano Spina, Lawrence Cavedon, Hideo Joho, and Mark Sanderson. 2018. Informing the design of spoken conversational search: Perspective paper. In Proceedings of the 2018 conference on human information interaction & retrieval, pages 32–41. find
  54. Ellen M Voorhees et al. 1999. The trec-8 question answering track report. In Trec, volume 99, pages 77–82.
  55. Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tie-Yan Liu. 2013. A theoretical analysis of ndcg type ranking measures. In Conference on learning theory, pages 25–54. PMLR. ↗
  56. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-ofthe-art natural language processing. arXiv preprint arXiv:1910.03771. ↗
  57. Jingjing Xu, Yuechen Wang, Duyu Tang, Nan Duan, Pengcheng Yang, Qi Zeng, Ming Zhou, and Xu Sun. 2019. Asking clarification questions in knowledge-based question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1618–1629. ↗
  58. Xin Yan and Xiaogang Su. 2009. Linear regression analysis: theory and computing. world scientific. ↗
  59. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32. ↗
  60. Munazza Zaib, Wei Emma Zhang, Quan Z Sheng, Adnan Mahmood, and Yang Zhang. 2021. Conversational question answering: A survey. arXiv preprint arXiv:2106.00874. ↗
  61. Hamed Zamani, Gord Lueck, Everest Chen, Rodolfo Quispe, Flint Luu, and Nick Craswell. 2020. Mimics: A large-scale data collection for search clarification. In Proceedings of the 29th acm international conference on information & knowledge management, pages 3189–3196. ↗
  62. Hamed Zamani, Johanne R Trippas, Jeff Dalton, and Filip Radlinski. 2022. Conversational information seeking. arXiv preprint arXiv:2201.08808. ↗
  63. Zhiling Zhang and Kenny Zhu. 2021. Diverse and specific clarification question generation with keywords. In Proceedings of the Web Conference 2021, pages 3501–3511. find
  64. Ziliang Zhao, Zhicheng Dou, Jiaxin Mao, and Ji-Rong Wen. 2022. Generating clarifying questions with web search results. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. find
  65. Jie Zou, Aixin Sun, Cheng Long, Mohammad Aliannejadi, and Evangelos Kanoulas. 2023. Asking clarifying questions: To benefit or to disturb users in web search? Information Processing & Management, 60(2):103176. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rahmani-2023-survey-asking-clarification-questions-datasets/