Elamur Research
☰
home / papers / 2019 / DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension

DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/Q19-1014.pdf

Summary

Most machine reading comprehension benchmarks draw on formal written sources such as news, Wikipedia or fiction, so conversational text has received little attention. Sun et al. introduce DREAM, a multiple-choice benchmark built from dialogues in English-as-a-Foreign-Language exams that teachers prepared for Chinese learners. Each question has three options, and exactly one is correct. The conversations cover everyday settings such as shops, airports, classrooms and phone calls.

The data are demanding. Most correct options are not copied from the dialogue, most questions need evidence spread across several sentences or turns, and about a third depend on commonsense that the speakers never state. Neural readers built for written text barely beat a simple lexical matcher. The authors then add speaker-aware dialogue structure and world knowledge (ConceptNet, PMI, a pre-trained language model) to rule-based, gradient-boosting and fine-tuned language model systems, which yields clear gains. Even so, the best system remains far below human accuracy. This makes DREAM a useful and difficult testbed for dialogue understanding.

Key points

  • Data set size and structure: DREAM contains 10,197 questions over 6,444 multi-turn, multi-party dialogues, with three options per question. Dialogues average 4.7 turns (maximum 48) and have up to 7 speakers. The data set is available at dataset.org/dream.
  • Difficulty profile: About 84% of correct options are non-extractive (83.7%). Some 85% of questions require multi-sentence reasoning, and 34% require commonsense knowledge.
  • Baselines: The best non-dialogue neural model reaches 45.5% accuracy. A lexical baseline (Richardson et al., 2013) reaches 44.6%.
  • Enhanced models: The improved systems are DSW++, GBDT++ and FTLM++, which add speaker information plus ConceptNet, PMI and fine-tuned LM knowledge. They gain up to 11.9 points absolute over the model of Wang et al. (2018b). The best automated result is 59.5%, against a human ceiling of 98.6%.
  • Ablations (dev set):
    • Dropping speaker embeddings from FTLM++ costs 1.7 points.
    • Dropping ConceptNet embeddings from DSW++ costs 4.7 points.
    • Removing all knowledge features from GBDT++ costs 6.2 points.
    • FTLM++ degrades on dialogues longer than 10 turns.
    • Both FTLM++ and GBDT++ score below chance on arithmetic questions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 114 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00264

@article{sun2019dream,
  title = {{DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension}},
  author = {Sun, Kai and Yu, Dian and Chen, Jianshu and Yu, Dong and Choi, Yejin and Cardie, Claire},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.1162/tacl\_a\_00264},
  url = {https://doi.org/10.1162/tacl_a_00264}
}

References

51 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017. Attention-over-attention neural networks for reading comprehension. In Proceedings of the ACL, pages 593–602. Vancouver, Canada. find
  2. Leila Amgoud, Yannis Dimopoulos, and Pavlos Moraitis. 2007. A unified and general framework for argumentation-based negotiation. In Proceedings of the AAMAS, pages 1–8. New York, NY, USA. find
  3. Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers for text comprehension. In Proceedings of the ACL, pages 1832–1846. Vancouver, Canada. find
  4. Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2016. Embracing data abundance: Booktest data set for reading comprehension. CoRR, cs.CL/1610.00956v1. find
  5. Steven Bird and Edward Loper. 2004. NLTK: the natural language toolkit. In Proceedings of the ACL on Interactive poster and demonstration sessions, pages 31–34. Barcelona, Spain. find
  6. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. SearchQA: A new Q&A data set augmented with context from a search engine. CoRR, cs.CL/1704.05179v3. find
  7. Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of the ACL, pages 789–797. Columbus, OH. find
  8. Maxwell Forbes and Yejin Choi. 2017. Verb physics: Relative physical knowledge of actions and objects. In Proceedings of the ACL, pages 266–276. Vancouver, Canada. find
  9. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the CNN/Daily Mail reading comprehension task. In Proceedings of the ACL, pages 2358–2367. Berlin, Germany. find our card
  10. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the NIPS, pages 1693–1701. Montreal, Canada. find our card
  11. Yu-Hsin Chen and Jinho D. Choi. 2016. Character identification on multiparty conversation: Identifying mentions of characters in TV shows. In Proceedings of the SIGDial, pages 90–100. Los Angeles, CA. find
  12. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children’s books with explicit memory representations. In Proceedings of the ICLR. Caribe Hilton, Puerto Rico. find our card
  13. Zhipeng Chen, Yiming Cui, Wentao Ma, Shijin Wang, and Guoping Hu. 2019. Convolutional spatial attention model for reading comprehension with multiple-choice questions. In Proceedings of the AAAI. Honolulu, HI. find
  14. Danqing Huang, Jing Liu, Chin-Yew Lin, and Jian Yin. 2018. Neural math word problem solver with reinforcement learning. In Proceedings of the COLING, pages 213–223. Santa Fe, NM. find
  15. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. pages 2174–2184. Brussels, Belgium. find our card
  16. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge data set for reading comprehension. CoRR, cs.CL/1705.03551v2. find
  17. Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter D. Turney, and Daniel Khashabi. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In Proceedings of the AAAI, pages 2580–2586. Phoenix, AZ. find
  18. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the NAACL-HLT, pages 252–262. New Orleans, LA. find our card
  19. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gáabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association of Computational Linguistics, 6:317–328. find our card
  20. I Nation. 2006. How large a vocabulary is needed for reading and listening? Canadian Modern Language Review, 63:59–82. find
  21. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension data set. CoRR, cs.CL/1611.09268v2. find
  22. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale reading comprehension data set from examinations. In Proceedings of the EMNLP, pages 785–794. Copenhagen, Denmark. find
  23. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did What: A large-scale person-centered cloze data set. In Proceedings of the EMNLP, pages 2230–2235. Austin, TX. find
  24. Chia-Hsuan Lee, Shang-Ming Wang, Huan-Cheng Chang, and Hung-Yi Lee. 2018. ODSQA: Open-domain spoken question answering data set. CoRR, cs.CL/1808.02280v1. find
  25. Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018. SemEval-2018 Task 11: Machine comprehension using commonsense knowledge. In Proceedings of the SemEval, pages 747–757. New Orleans, LA. find
  26. Chia-Hsuan Li, Szu-Lin Wu, Chi-Liang Liu, and Hung-yi Lee. 2018. Spoken SQuAD: A study of mitigating the impact of speech recognition errors on listening comprehension. CoRR, cs.CL/1804.00320v1. find
  27. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Eduard Duchesnay. 2011. Scikit-learn:… find
  28. Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating Wikipedia by summarizing long sequences. In Proceedings of the ICLR. Vancouver, Canada. find our card
  29. Peter LoBue and Alexander Yates. 2011. Types of common-sense knowledge needed for recognizing textual entailment. In Proceedings of the ACL, pages 329–334. Portland, OR. find
  30. Anselmo Penas, Yusuke Miyao, Alvaro Rodrigo, Eduard H Hovy, and Noriko Kando. 2014. Overview of CLEF QA Entrance Exams Task 2014. In Proceedings of the CLEF, pages 1194–1200. Sheffield, UK. find
  31. Kaixin Ma, Tomasz Jurczyk, and Jinho D. Choi. 2018. Challenging reading comprehension on daily conversation: Passage completion on multi-party dialog. In Proceedings of the NAACL-HLT, pages 2039–2048. New Orleans, LA. find
  32. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the NAACL-HLT, pages 2227–2237, New Orleans, LA. find
  33. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new data set for open book question answering. In Proceedings of the EMNLP. Brussels, Belgium. find
  34. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Preprint, available at https://openai.com/blog/language-unsupervised/. ↗
  35. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluvation framework for deeper understanding of commonsense stories. In Proceedings of the NAACL-HLT, pages 839–849. San Diego, CA. find
  36. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the EMNLP, pages 2383–2392. find our card
  37. Austin, TX. and Kaheer Suleman. 2017. NewsQA: A machine comprehension data set. In Proceedings of the RepL4NLP, pages 191–200. Vancouver, Canada. find
  38. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. CoQA: A conversational question answering challenge. CoRR, cs.CL/1808. 07042v1. find our card
  39. Bo-Hsiang Tseng, Sheng-Syun Shen, Hung-Yi Lee, and Lin-Shan Lee. 2016. Towards machine comprehension of spoken content: Initial toefl listening comprehension test by machine. In Proceedings of the Interspeech. San Francisco, CA. find
  40. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. MCTest: A challenge data set for the open-domain machine comprehension of text. In Proceedings of the EMNLP, pages 193–203. Seattle, WA. find
  41. Lei Wang, Yan Wang, Deng Cai, Dongxiang Zhang, and Xiaojiang Liu. 2018a. Translating a math word problem to a expression tree. In Proceedings of the EMNLP, pages 1064–1069. Brussels, Belgium. find
  42. Hideyuki Shibuki, Kotaro Sakamoto, Yoshinobu Kano, Teruko Mitamura, Madoka Ishioroshi, Kelly Y Itakura, Di Wang, Tatsunori Mori, and Noriko Kando. 2014. Overview of the NTCIR-11 QA-Lab Task. In NTCIR. find
  43. Shuohang Wang, Mo Yu, Shiyu Chang, and Jing Jiang. 2018b. A co-matching model for multi-choice reading comprehension. In Proceedings of the ACL, pages 1–6. Melbourne, Australia. find
  44. Richard Sikes. 2007. Fuzzy matching in theory and practice. Multilingual, 18(6):39–43. find
  45. Wen-tau Yih, Ming-Wei Chang, Christopher Meek, and Andrzej Pastusiak. 2013. Question answering using enhanced lexical semantic models. In Proceedings of the ACL, pages 1744–1753. Sofia, Bulgaria. find
  46. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. In Proceedings of the AAAI, pages 4444–4451. San Francisco, CA. find
  47. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. QANet: Combining local convolution with global self-attention for reading comprehension. In Proceedings of the ICLR. Vancouver, Canada. find our card
  48. Chenhao Tan and Lillian Lee. 2015. All who wander: On the prevalence and characteristics of multi-community engagement. In Proceedings of the WWW, pages 1056–1066. Florence, Italy. find
  49. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE ICCV, pages 19–27. Santiago, Chile. find our card
  50. Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Multi-range reasoning for machine comprehension. CoRR, cs.CL/1803.09074v1. find
  51. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, find