Elamur Research⌕
☰
home / papers / 2023 / InSCIt: Information-Seeking Conversations with Mixed-Initiative Interactions

InSCIt: Information-Seeking Conversations with Mixed-Initiative Interactions

Paper · 2023 · Zeqiu Wu, Ryu Parish, Hao Cheng et al.

Summary

Conversational QA systems usually assume each user question has one direct answer in a single passage. Real users, however, often ask vague or unanswerable questions, and a helpful agent should sometimes take the lead by asking for clarification or offering related partial information. Existing datasets rarely capture this agent-side initiative, or simulate it artificially. The authors built InSCIt, a corpus of natural human-human dialogues grounded in Wikipedia. Crowdworkers played users, agents and validators in a turn-by-turn pipeline. Agents searched Wikipedia, chose a response strategy and marked the passages supporting their reply. The dataset defines two tasks: identifying a set of evidence passages and generating a grounded response. It also comes with a human evaluation protocol covering coherence, factual consistency and comprehensiveness. Two baselines were tested, one built on open-domain QA models (DPR/FiD-style) and one pipelining DIALKI with FiD, which performed best. Both remain far below human agents, especially when no direct answer exists. This makes InSCIt a useful benchmark for mixed-initiative, multi-passage dialogue.

Key points

  • InSCIt contains 805 conversations with about 4.7K user-agent turns. The knowledge source is the full Wikipedia dump of 04/20/2022, and data and code are released on GitHub (ellenmellon/INSCIT).
  • Agent turns break down into direct answers (~72%), clarification questions (~13%) and relevant partial information (~13%). Many turns rely on more than one evidence passage.
  • Seed articles were sampled from five topic areas: food and drink, hobbies, historical events, geography, and Wikipedia's weekly top-25 pages.
  • In the response-type experiment, the classifier reached 0.75 accuracy against 0.73 for always predicting "direct answer". Feeding FiD the oracle response type raised test scores from BLEU 30.6 / RG-F1 44.3 to 31.6 / 47.4.
  • The main open challenges are better passage selection, fusing evidence from several passages, and learning when to use non-direct strategies. Transfer from other QA resources is suggested to offset the small training set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓University of WashingtonSeattle · United States353 works · 200,043 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States275 works · 97,634 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Zeqiu Wu, Ryu Parish, Hao Cheng, Sewon Min, Prithviraj Ammanabrolu, Mari Ostendorf and Hannaneh Hajishirzi. 2023. InSCIt: Information-Seeking Conversations with Mixed-Initiative Interactions. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00559

@article{wu2023inscit,
  title = {{InSCIt: Information-Seeking Conversations with Mixed-Initiative Interactions}},
  author = {Zeqiu Wu, Ryu Parish, Hao Cheng, Sewon Min, Prithviraj Ammanabrolu, Mari Ostendorf and Hannaneh Hajishirzi},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00559},
  url = {https://doi.org/10.1162/tacl_a_00559}
}

References

33 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman, Harm de Vries, and Siva Reddy. 2022. TopiOCQA: Open-domain conversational question answering with topic switching. Transactions of the Association for Computational Linguistics, 10:468–483. https://doi.org/10.1162/tacl a 00471 ↗
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  3. Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2021. Building and evaluating open-domain dialogue corpora with clarifying questions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4473–4484, Online and Punt… find
  4. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.
  5. Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM. https://doi .org/10.114… find
  6. Song Feng, Siva Patel, and Hui Wan. 2022. find
  7. DialDoc 2022 shared task: Open-book document-grounded dialogue modeling. In Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering, pages 155–160, Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Dublin, Ir…
  8. Song Feng, Siva Sankalp Patel, Hui Wan, and Sachindra Joshi. 2021. MultiDoc2Dial: Modeling dialogues grounded in multiple documents. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6162–6176, Online and Punta Cana, Dominican Republic. Association for… find
  9. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli…
  10. Song Feng, Hui Wan, Chulaka Gunasekara, Siva Patel, Sachindra Joshi, and Luis Lastras. 2020. doc2dial: A goal-oriented document-grounded dialogue dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8118–8128, Online. Association for Compu…
  11. Natalia Konstantinova and Constantin Orasan. 2013. Interactive question answering. Emerging Applications of Natural Language Processing: Concepts and New Research, pages 149–169. https://doi.org/10.4018/978-1-4666 -2169-5.ch007 — [doi:10.4018/978-1-4666](https://doi.org/10.4018/978-1-4666) ↗
  12. Chang Gao, Wenxuan Zhang, and Wai Lam. 2022. UniGDD: A unified generative framework for goal-oriented document-grounded dialogue. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 599–605, Dublin, Ireland. Association for Compu… find
  13. Huihan Li, Tianyu Gao, Manan Goenka, and Danqi Chen. 2022. Ditch the gold standard: Re-evaluating conversational question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8074–8085, Dublin, Ireland. Association for C… find
  14. Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, pages 5110–5117. https://doi.org/10.1609/aaai.v32i1 .1197… — [doi:10.1609/aaai.v32i1](https://doi.org/10.1609/aaai.v32i1) ↗
  15. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783–5797, Online. Association for Computational Linguistics. https… find
  16. Meiqi Guo, Mingda Zhang, Siva Reddy, and Malihe Alikhani. 2021. Abg-coQA: Clarifying ambiguity in conversational question answering. In 3rd Conference on Automated Knowledge Base Construction. find
  17. Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. 2018.Towards exploiting background knowledge for building conversation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2322–2332, Brussels, Belgium. Association for Computa…
  18. Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength natural language processing in Python. find
  19. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin…
  20. Haitian Sun, William Cohen, and Ruslan Salakhutdinov. 2022. ConditionalQA: A complex reading comprehension dataset with conditional answers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3627–3637, Dublin, Ireland. Associati… find
  21. Gustavo Penha, Alexandru Balan, and Claudia Hauff. 2019. Introducing mantis: A novel multi-domain information seeking dialogues dataset. arXiv preprint arXiv:1912.04639. — [arXiv:1912.04639](https://arxiv.org/abs/1912.04639) find
  22. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics. https://doi.org/10.18653/v1/W18 -6319 ↗
  23. N. Webb and B. Webber. 2009. Special issue on interactive question answering: Introduction. Natural Language Engineering, 15(1):1–8. https://doi.org/10.1017/S1351324908004877 ↗
  24. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. https://doi.org/10.1162/tacl_a_00266 ↗
  25. Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computati…
  26. V. Rieser and O. Lemon. 2009. Does this list contain what you were searching for? Learning adaptive dialogue strategies for interactive question answering. Natural Language Engineering, 15(1):55–72. https://doi.org/10 .1017/S1351324908004907 find
  27. Zeqiu Wu, Bo-Ru Lu, Hannaneh Hajishirzi, and Mari Ostendorf. 2021. DIALKI: Knowledge identification in conversational systems through dialogue-document contextualization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1852–1863, Online and Punta Cana… find
  28. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389. https://doi.org/10.1561/1500000019 ↗
  29. Pedro Rodriguez, Paul Crook, Seungwhan Moon, and Zhiguang Wang. 2020. Information seeking in the spirit of learning: A dataset for conversational curiosity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8153–8172, Online. Association for Com… find
  30. Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’17, pages 1253–1256, New York, NY, USA. Association for Comp… find
  31. Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. find
  32. Michael Zhang and Eunsol Choi. 2021. Situated-QA: Incorporating extra-linguistic contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371–7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. https:/… find
  33. Kangyan Zhou, Shrimai Prabhumoye, and Alan W. Black. 2018. A dataset for document grounded conversations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 708–713, Brussels, Belgium. Association for Computational Linguistics. https://doi.org/10 .18653/…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2023-inscit-information-seeking-conversations-mixed/