Elamur Research
☰
home / papers / 2018 / QuAC: Question Answering in Context

QuAC: Question Answering in Context

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1241.pdf

Summary

QuAC is a large crowdsourced resource for machine reading comprehension in dialogs where a learner seeks information. Two Amazon Mechanical Turk workers are paired. A "student" sees only an article's opening paragraph and a section heading from a Wikipedia biography. A "teacher" sees the full section and answers each question by highlighting a text span, or by marking it unanswerable. The teacher also attaches dialog acts that signal yes/no, answerability, and whether the student should pursue the current thread. The student cannot see the evidence, so the questions tend to be exploratory, context-dependent, and elliptical, with less surface overlap with the text than in SQuAD or CoQA. The authors tested several baselines, including BiDAF++ (Clark and Gardner, 2018) augmented with previous dialog turns. Even the strongest version trails human agreement by roughly 20 F1. QuAC therefore offers a demanding benchmark for conversational QA systems that must track discourse history.

Key points

  • Size: 13,594 dialogs with 98,407 questions over 8,854 Wikipedia sections. About 20% of questions have no answer, and roughly a quarter are yes/no.
  • Answer format: Answers are spans of at most 30 tokens. They average over 14 tokens, compared with about 3 tokens in SQuAD and CoQA. Each evaluation question received four extra reference annotations.
  • Dialog acts: Teachers label three dialog act types: continuation (follow up / maybe / don't), affirmation, and answerability.
  • Best model: BiDAF++ with 2 turns of context. Its performance reaches the human level for just 60% of individual questions and a mere 5% of full dialogs.
  • Error analysis: Context-aware models degrade less over later turns and handle small topic shifts better. All models struggle with short answers and large jumps in the text. Data, baselines, and a leaderboard are available at quac.ai.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Speech act theory

Cited by

389 papers of our library cite it (of the 140673 whose bibliographies we read); 29 of them have a page here, 360 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1241 · arXiv 1808.07036

@inproceedings{choi2018quac,
  title = {{QuAC: Question Answering in Context}},
  author = {Choi, Eunsol and He, He and Iyyer, Mohit and Yatskar, Mark and Yih, Wen-tau and Choi, Yejin and Liang, Percy and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1241},
  eprint = {1808.07036},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d18-1241}
}

References

29 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. Proceedings of the Association for Computational Linguistics. find our card
  2. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  3. Deborah A. Dahl, Madeleine Bates, Michael Brown, William M. Fisher, Kate Hunicke-Smith, David S. Pallett, Christine Pao, Alexander I. Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the atis task: The atis-3 corpus. In Proceedings of the workshop on Human Language Technology. find
  4. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Computer Vision and Pattern Recognition. find
  5. Hao Fang, Hao Cheng, Elizabeth Clark, Ariel Holtzman, Maarten Sap, Mari Ostendorf, Yejin Choi, and Noah A Smith. 2017. Sounding board–university of washingtons alexa prize submission. Alexa Prize Proceedings. find
  6. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: A deep semantic natural language processing platform. find our card
  7. Tomás Kociský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2017. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, abs/1712.07040. find our card
  8. John Langford, Lihong Li, and Alex Strehl. 2007. Vowpal wabbit online learning project. find
  9. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke S. Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Conference on Computational Natural Language Learning. find
  10. Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning for negotiation dialogues. Proceedings of Empirical Methods in Natural Language Processing. find
  11. Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016. Deep reinforcement learning for dialogue generation. Proceedings of Empirical Methods in Natural Language Processing. find our card
  12. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv, abs/1611.09268. find our card
  13. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. Conference of the North American Chapter of the Association for Computational Linguistics. find
  14. Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. Association for the Advancement of Artificial Intelligence. find
  15. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the Association for Computational Linguistics. find our card
  16. He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. Proceedings of the Association for Computational Linguistics. find
  17. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  18. Mohit Iyyer, Wen tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for sequential question answering. In Proceedings of the Association for Computational Linguistics. find our card
  19. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. Coqa: A conversational question answering challenge. ArXiv. find our card
  20. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the Association for Computational Linguistics. find our card
  21. Tomasz Jurczyk, Amit Deshmane, and Jinho D. Choi. 2018. Analysis of wikipedia-based corpora for question answering. arXiv preprint arXiv:abs/1801.02073. — [arXiv:1801.02073](https://arxiv.org/abs/1801.02073) find
  22. Alan Ritter, Colin Cherry, and William B. Dolan. 2011. Data-driven response generation in social media. In Proceedings of Empirical Methods in Natural Language Processing. find
  23. Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Signh, Tim Rocktschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. In Proceedings of Empirical Methods in Natural Language Processing. find
  24. Amrita Saha, Vardaan Pahuja, Mitesh M Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowledge graph. In Association for the Advancement of Artificial Intelligence. find
  25. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. Proceedings of the International Conference on Learning Representations. find our card
  26. Manfred Stede and David Schlangen. 2004. Information-seeking chat: Dialogues driven by topic-structure. In Eighth workshop on the semantics and pragmatics of dialogue; SemDial. find
  27. Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In Proceedings of the World Wide Web Conference. find
  28. A. Talmor and J. Berant. 2018. The web as knowledgebase for answering complex questions. In Conference of the North American Chapter of the Association for Computational Linguistics. find
  29. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830. find our card