Elamur Research⌕
☰
home / papers / 2021 / Open-Domain Question Answering Goes Conversational via Question Rewriting

Open-Domain Question Answering Goes Conversational via Question Rewriting

Summary

Existing conversational QA benchmarks such as QuAC and CoQA confine each dialogue to a single document. In real use, a conversation can drift between topics and its answers may sit on many different web pages. To address this, the authors built QReCC, an open-domain conversational QA dataset of about 13.6K dialogues and 80K question-answer pairs. Professional annotators wrote each dialogue alone, starting from seed questions taken from QuAC, TREC CAsT and Natural Questions. They searched the web for answers, wrote a self-contained rewrite for every question, and linked each answer to its source page. The answer collection includes these pages plus Common Crawl data, giving roughly 10M pages split into 54M passages.

The baseline handles conversation by first rewriting each question so it stands alone. It then applies a standard open-domain QA pipeline, BERTserini, which combines BM25 retrieval with a BERT reader. The best rewriting model, Transformer++, roughly doubles the end-to-end F1 obtained with the original questions. Even so, it reaches only 19.10 F1, against an extractive upper bound of 75.45. This leaves substantial room for better retrieval and for abstractive answer generation.

Key points

  • Dataset size and splits: QReCC has 80.0K questions in 13.6K dialogues, averaging 6 turns. The splits are 50.8K train, 12.7K dev and 16.4K test questions. It is released at github.com/apple/ml-qrecc.
  • Coverage and annotation: It is the first large-scale resource covering question rewriting, passage retrieval and reading comprehension together. Annotation used a self-dialog setup with 30 professional annotators, followed by a 5-reviewer quality check.
  • Question rewriting results: All generative rewriting models beat the AllenAI Coref coreference baseline, and Transformer++ performed best. Retrieval MRR was 0.0343 with original questions, 0.1586 with Transformer++ rewrites and 0.1994 with human rewrites.
  • End-to-end QA results (F1): Original questions scored 9.07, Transformer++ rewrites 19.10 and human rewrites 21.82. Given the gold passage ("Known Context"), Transformer++ reaches 32.34 F1.
  • Upper bound and answer style: The extractive upper bound is 75.45 F1 (25.07 EM). Some reference answers paraphrase or combine content from several passages, so span extraction alone cannot fully match them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Question answering · Dialogue systems

Institutions

🎓University of AmsterdamAmsterdam · Netherlands71 works · 79 805 citations🏢Apple (Israel)Herzliya · Israel3 works · 789 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.naacl-main.44

@inproceedings{anantha2021open,
  title = {{Open-Domain Question Answering Goes Conversational via Question Rewriting}},
  author = {Anantha, Raviteja and Vakulenko, Svitlana and Tu, Zhucheng and Longpre, Shayne and Pulman, Stephen and Chappidi, Srinivas},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.44},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.44}
}

References

33 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. Ms marco: A human generated machine reading comprehension dataset. In Advances in N… find our card
  2. Bill Byrne, Karthik Krishnamoorthi, Sankar Chinnadhurai, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Andy Cedilnik, and Kim Kyu-Young. 2019. Taskmaster-1: Toward a realistic and diverse dialog dataset. In Proceedings of the 2019 Conference on Empirical Methods in Nat… find our card
  3. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for english. In Proceedings of the 2018 Conference on Empirical Methods in Natural Langua… find our card
  4. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870– 1879. find our card
  5. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. Quac: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 217… find our card
  6. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen,… find our card
  7. Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2019. find
  8. Cast 2019: The conversational assistance track overview. In Proceedings of the Twenty-Eighth Text REtrieval Conference, TREC, pages 13–15. find
  9. Michael J. Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 26-27, 2014, Baltimore, Maryland, USA, pages 376–380. find our card
  10. Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019. Can you unpack that? learning to rewrite questions-in-context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5920–5… find our card
  11. Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109. find our card
  12. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… find our card
  13. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096. find our card
  14. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana,… find our card
  15. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81. find our card
  16. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. Coqa: A conversational question answering challenge. TACL, 7:249–266. find our card
  17. Sheng-Chieh Lin, Jheng-Hong Yang, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy Lin. 2020. Query reformulation using query history for passage retrieval in conversational search. CoRR, abs/2005.02230. — [arXiv:2005.02230](https://arxiv.org/abs/2005.02230) find
  18. Pengjie Ren, Zhumin Chen, Zhaochun Ren, Evangelos Kanoulas, Christof Monz, and Maarten de Rijke. 2020. Conversations with search engines. find
  19. Shayne Longpre, Yi Lu, Zhucheng Tu, and Chris DuBois. 2019. An exploration of data augmentation and sampling techniques for domain-agnostic question answering. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 220–227. find
  20. Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processi… find our card
  21. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. Association for Computational Linguistics, pages 280–290. find
  22. Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu, and Raviteja Anantha. 2020. Question rewriting for conversational question answering. CoRR, abs/2004.14652. — [arXiv:2004.14652](https://arxiv.org/abs/2004.14652) find
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318. find our card
  24. Nikos Voskarides, Dan Li, Pengjie Ren, Evangelos Kanoulas, and Maarten de Rijke. 2020. Query resolution for conversational search with limited supervision. In SIGIR 2020: 43rd international ACM SI-GIR conference on Research and Development in Information Retrieval. find
  25. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and… find our card
  26. Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W. Bruce Croft, and Mohit Iyyer. 2020. Open-retrieval conversational question answering. In SI-GIR 2020: 43rd international ACM SIGIR conference on Research and Development in Information Retrieval. find
  27. Jun Quan, Deyi Xiong, Bonnie Webber, and Changjian Hu. 2019. Gecor: An end-to-end generative ellipsis and co-reference resolution model for task-oriented dialogue. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on… find our card
  28. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  29. Nick Webb, editor. 2006. Proceedings of the Interactive Question Answering Workshop at HLT-NAACL 2006. Association for Computational Linguistics, New York, NY, USA. find
  30. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019a. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card
  31. Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019b. Transfertransfo: A transfer learning approach for neural network based conversational agents. CoRR, abs/1901.08149. — [arXiv:1901.08149](https://arxiv.org/abs/1901.08149) find
  32. Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1253–1256. find
  33. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with bertserini. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 72… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/anantha-2021-open-domain-question-answering-goes/