Elamur Research⌕
☰
home / papers / 2022 / TopiOCQA: Open-domain Conversational Question Answering with Topic Switching

TopiOCQA: Open-domain Conversational Question Answering with Topic Switching

Paper · 2022 · Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman et al.

Summary

Benchmarks for conversational question answering (CQA), such as CoQA and QuAC, give the system a fixed reference passage and keep each dialogue on a single subject. Real search sessions are different: the relevant text must be found first, and users drift between related topics. The authors address both gaps with TopiOCQA (pronounced "Tapioca"), an open-domain CQA dataset built on a 5.9-million-document Wikipedia snapshot in which every article counts as one topic. Each dialogue was produced by two annotators. The questioner saw only article and section titles. The answerer read full articles, followed hyperlinks, and replied in free-form text. Most conversations start from a Natural Questions query and then move to linked subjects. Baselines combine sparse (BM25) or dense (DPR) retrieval with an extractive (DPR Reader) or generative (Fusion-in-Decoder) reader. The question is fed as raw text, with the full dialogue history, or as a standalone rewrite. The best system still trails humans by 14.2 F1, and zero-shot GPT-3 lags much further. This makes the dataset a demanding testbed for agents that must re-retrieve whenever the topic shifts.

Key points

  • TopiOCQA has 3,920 conversations and 50,574 QA pairs. Dialogues average about 13 turns and touch roughly 4 Wikipedia documents, and 28% of turns need a different document than the previous turn.
  • Seed topics come from about 35k Wikipedia "good" articles (dump of 10/20/2020), while later turns can use any page.
  • The top pipeline is DPR Retriever + FiD with the full-history input. It scores 33.4 EM / 55.8 F1 on test, which is 6.9 EM and 14.2 F1 below humans. GPT-3 in a closed-book zero-shot setting reaches 31.8 F1 and often hallucinates answers.
  • Full history works better with the dense DPR retriever, and rewrites work better with BM25. Using only the original question is consistently the weakest option.
  • Readers fall short even with retrieval removed. When given the gold passage, FiD still misses human F1 by 5.9 points. The extractive upper bound is 47.3 EM / 81.0 F1, and DPR Reader stays far below it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓McGill UniversityMontreal · Canada51 works · 10,476 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada31 works · 13,061 citations🎓ETH ZurichZurich · Switzerland36 works · 4,534 citations🔬Canadian Institute for Advanced ResearchToronto · Canada17 works · 16,274 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

🧭 Topics 2Research subjects and languages the work is about.Question answering · 242 worksDialogue systems · 401 works

Cite

Cite as Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman, Harm de Vries and Siva Reddy. 2022. TopiOCQA: Open-domain Conversational Question Answering with Topic Switching. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00471

@article{adlakha2022topiocqa,
  title = {{TopiOCQA: Open-domain Conversational Question Answering with Topic Switching}},
  author = {Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman, Harm {de Vries} and Siva Reddy},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.1162/tacl\_a\_00471},
  url = {https://doi.org/10.1162/tacl_a_00471}
}

References

40 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  2. Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021. Open-domain question answering goes conversational via question rewriting. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguist…
  3. Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019. Can you unpack that? Learning to rewrite questions-in-context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP…
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chr…
  5. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874–880, Online. https://doi.org /10.186… ↗
  6. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879, Vancouver, Canada.
  7. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. https://doi.org/10.18653/v1/D17-1215 ↗
  8. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, Brussels, Belgium.
  9. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli…
  10. Jeffrey Dalton, Chenyan Xiong, Vaibhav Kumar, and Jamie Callan. 2020. CAsT-19: A dataset for conversational information seeking. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, pages 1985–1988, New York, NY, USA. https://… — [doi:10.1145/3397271](https://doi.org/10.1145/3397271) find
  11. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The Narrative-QA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328. https://doi.org/10.1162/tacl_a_00023 ↗
  12. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural…
  13. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  14. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. https://doi.org/10.18653/v1/P18 -2… ↗
  15. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy.
  16. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. In CoCo@NIPS.
  17. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. https://doi.org/10.18653/v1/D16-1264 ↗
  18. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
  19. Siva Reddy, Danqi Chen, and Christopher Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7(0):249–266. https://doi.org/10.1162/tacl a 00266 ↗
  20. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. https://doi.org/10.18653/v1/2020.emnlp-main .437 ↗
  21. Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W. Bruce Croft, and Mohit Iyyer. 2020. Open-Retrieval Conversational Question Answering. SIGIR ’20, pages 539–548, New York, NY, USA. find
  22. Chen Qu, Liu Yang, Minghui Qiu, W. Bruce Croft, Yongfeng Zhang, and Mohit Iyyer. 2019a. BERT with history answer embedding for conversational question answering. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR’19, pages 1133–1… find
  23. Stephen Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. 1995. Okapi at TREC-3. In Overview of the Third Text Retrieval Conference (TREC-3), pages 109–126.
  24. Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2021. QA dataset explosion: A taxonomy of NLP resources for question answering and reading comprehension. arXiv preprint arXiv:2107.12708. find
  25. Chen Qu, Liu Yang, Minghui Qiu, Yongfeng Zhang, Cen Chen, W. Bruce Croft, and Mohit Iyyer. 2019b. Attentive history selection for conversational question answering. In Proceedings of the 28th ACM International Conference Harvey Sacks and Gail Jefferson. 1995. Lectures on conversation, Volume 1. page… find
  26. Anshumali Shrivastava and Ping Li. 2014. Asymmetric LSH (ALSH) for sublinear time maximum inner product search (MIPS). In Proceedings of the 27th International Conference on Neural Information Processing Systems -Volume 2, pages 2321–2329, Montreal, Canada. find
  27. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567. https://doi.org/10.18653/v1/2021 .findings-emnlp.320 ↗
  28. Amanda Spink, H. Cenk Ozmutlu, and Seda Ozmutlu. 2002. Multitasking information seeking and searching processes. Journal of the American Society for Information Science and Technology, 53(8):639–652. https://doi.org/10.1002/asi.10124 — [doi:10.1002/asi.10124](https://doi.org/10.1002/asi.10124) ↗
  29. Lin. 2019. End-to-end open-domain question answering with BERTserini. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 72–77, Minneapolis, Minnesota. https://doi.org/10.18653/v1/N19-4013 ↗
  30. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018, Lisbon, Portugal. https://doi.org/10.18653/v1/D15-1237 ↗
  31. Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2019. SDNet: Contextualized attention-based deep network for conversational question answering. arXiv preprint arXiv:1812 .03593. find
  32. Manfred Stede and David Schlangen. 2004. Information-seeking chat: Dialogue management by topic structure. find
  33. A Annotators Details Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu, and Raviteja Anantha. 2021. Question rewriting for conversational question answering. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM ’21, pages 355–363, New York, NY, USA. https://doi.o… ↗
  34. Each conversation in T OPI OCQA is an interaction between two annotators, a questioner and an answerer. The annotators were selected from TAKT’s in-house workforce, based on their English language proficiency and trained for the role of both questioner and answerer. The annotators are provided with… find
  35. Douglas Walton. 2019. The New Dialectic: Conversational Contexts of Argument. University of Toronto Press. find
  36. Guidelines for the Questioner: • The first question should be unambiguous and about the seed entity. find
  37. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018. R3: Reinforced ranker-reader for open-domain question answering. In AAAI, pages 5981–5988. find
  38. The follow-up questions be contextualized and dependent on the conversation history whenever possible. • Avoid using same words as in section titles of the document. For example, if the section title is ‘‘Awards’’, a plausible question can be ‘‘What accolades did she receive for her work?’’. find
  39. Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019. Multi-passage BERT: A globally normalized BERT model for open-domain question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference o…
  40. The conversation should involve multiple documents (topics). Guidelines for the Answerer: • Based on the question, identify the relevant document and section. • The answer should be based on the contents of the identified document. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong,… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/adlakha-2022-topiocqa-open-domain-conversational-question/