Elamur Research⌕
☰
home / papers / 2019 / Improving Machine Reading Comprehension with General Reading Strategies

Improving Machine Reading Comprehension with General Reading Strategies

Paper · 2019 · Kai Sun, Dian Yu, Dong Yu et al.

Summary

The paper addresses non-extractive, multiple-choice machine reading comprehension (MRC), where many correct answers are not spans of the source text. Labeled data for these tasks is scarce, and adding knowledge through further pre-training is very expensive. The authors therefore ask whether a fixed pre-trained model can be improved during fine-tuning alone. Drawing on reading strategies from cognitive science, they add three techniques to fine-tuning of the GPT transformer (Radford et al., 2018):

  • Back and Forth Reading (BF): combines models fed the input in original and reversed segment order.
  • Highlighting (HL): adds a learned embedding to document tokens that relate to the question and answer options.
  • Self-Assessment (SA): creates practice questions and options from the documents without supervision.

The model is first fine-tuned on RACE and then on a target dataset. With these strategies, accuracy on RACE rises by 5.8 points over plain GPT fine-tuning. The approach also sets new state-of-the-art results on six diverse benchmarks. This shows that low-cost, domain-independent changes to fine-tuning can substitute for some costly pre-training.

Key points

  • Source data is RACE, built from language exams, with 27,933 documents and 97,687 questions. It is the largest general-domain multiple-choice MRC dataset.
  • Evaluation covers six target datasets: MCTest, MultiRC, ARC, SemEval-2018 Task 11, OpenBookQA, and ROCStories.
    • MultiRC allows several correct options, so it is handled with a sigmoid output.
    • ROCStories has no explicit questions, so the question slot is left empty.
  • Average accuracy improves by 6.2 points over the previous SOTA (60.1%) and by 7.8 points over an ensemble baseline without strategies (58.5%).
  • When RACE labels are not used, the strategies still raise average accuracy from 54.6% to 60.3%. Gains are largest on ARC, OpenBookQA and MCTest.
  • Core code is available at github.com/nlpdata/strategy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cornell UniversityIthaca · United States89 works · 58,422 citations🏢TencentShenzhen · China66 works · 17,945 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksDevelopmental and Educational Psychology · 260 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 242 works

Cite

Cite as Kai Sun, Dian Yu, Dong Yu and Claire Cardie. 2019. Improving Machine Reading Comprehension with General Reading Strategies. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1270

@inproceedings{sun2019improving,
  title = {{Improving Machine Reading Comprehension with General Reading Strategies}},
  author = {Kai Sun, Dian Yu, Dong Yu and Claire Cardie},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1270},
  url = {https://doi.org/10.18653/v1/n19-1270}
}

References

60 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2016. Embracing data abundance: Booktest dataset for reading comprehension. CoRR, cs.CL/1610.00956v1. find
  2. Florian Boudin, Hugo Mougard, and Benoit Favre. 2015. Concept-based summarization using integer linear programming: From concept pruning to multiple optimal solutions. In Proceedings of the EMNLP, pages 17–21, Lisbon, Portugal. find
  3. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the CNN/Daily Mail reading comprehension task. In Proceedings of the ACL, pages 2358–2367, Berlin, Germany.
  4. Yu-Hsin Chen and Jinho D Choi. 2016. Character identification on multiparty conversation: Identifying mentions of characters in TV shows. In Proceedings of the SIGDial, pages 90–100, Los Angeles, CA. find
  5. Zhipeng Chen, Yiming Cui, Wentao Ma, Shijin Wang, Ting Liu, and Guoping Hu. 2018. HFL-RC system at SemEval-2018 Task 11: Hybrid multi-aspects model for commonsense reading comprehension. CoRR, cs.CL/1803.05655v1. find
  6. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the EMNLP, pages 2174– 2184, Brussels, Belgium.
  7. Yu-An Chung, Hung-yi Lee, and James Glass. 2018. Supervised and unsupervised transfer learning for question answering. In Proceedings of the NAACL-HLT, pages 1585–1594, New Orlean, LA. find
  8. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. CoRR, cs.CL/1803.05457v1. find
  9. Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter D Turney, and Daniel Khashabi. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In Proceedings of the AAAI, pages 2580– 2586, Phoenix, AZ. https://arxiv.org/abs/1810.13441v1. — [arXiv:1810.13441](https://arxiv.org/abs/1810.13441) ↗
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL-HLT, Minneapolis, MN.
  11. Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers for text comprehension. In Proceedings of the ACL, pages 1832–1846, Vancouver, Canada. find
  12. Peng Ding and Xiaobing Zhou. 2018. Ynu deep at semeval-2018 task 12: A bilstm model with neural attention for argument reasoning comprehension. In Proceedings of The SemEval, pages 1120–1123, New Orleans, LA. find
  13. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new Q&A dataset augmented with context from a search engine. CoRR, cs.CL/1704.05179v3.
  14. David Golub, Po-Sen Huang, Xiaodong He, and Li Deng. 2017. Two-stage synthesis networks for transfer learning in machine comprehension. CoRR, cs.CL/1706.09789v3. find
  15. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the NIPS, pages 1693–1701, Montreal, Canada.
  16. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children’s books with explicit memory representations. In Proceedings of the ICLR, Caribe Hilton, Puerto Rico.
  17. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. CoRR, cs.CL/1705.03551v2.
  18. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the NAACL-HLT, pages 252–262, New Orleans, LA.
  19. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gáabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association of Computational Linguistics, 6:317–328.
  20. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale reading comprehension dataset from examinations. In Proceedings of the EMNLP, pages 785–794, Copenhagen, Denmark.
  21. Yongbin Li and Xiaobing Zhou. 2018. Lyb3b at semeval-2018 task 11: Machine comprehension task using deep learning models. In Proceedings of the SemEval, pages 1073–1077, New Orleans, LA. find
  22. Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating Wikipedia by summarizing long sequences. In Proceedings of the ICLR, Vancouver, Canada.
  23. Kaixin Ma, Tomasz Jurczyk, and Jinho D Choi. 2018. Challenging reading comprehension on daily conversation: Passage completion on multiparty dialog. In Proceedings of the NAACL-HLT, pages 2039– 2048, New Orleans, LA. find
  24. Michael McCandless, Erik Hatcher, and Otis Gospodnetic. 2010. Lucene in Action, Second Edition: Covers Apache Lucene 3.0. Manning Publications Co., Greenwich, CT. find
  25. Danielle S McNamara, Irwin B Levinstein, and Chutima Boonthum. 2004. iSTART: Interactive strategy training for active reading and thinking. Behavior Research Methods, Instruments, & Computers, 36(2):222–233. find
  26. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the EMNLP, pages 2381–2391, Brussels, Belgium. find
  27. Jianmo Ni, Chenguang Zhu, Weizhu Chen, and Julian McAuley. 2018. Learning to attend on essential terms: An enhanced retriever-reader model for open-domain question answering. In Proceedings of the NAACL-HLT, Minneapolis, MN. find
  28. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did What: A large-scale person-centered cloze dataset. In Proceedings of the EMNLP, pages 2230–2235, Austin, TX.
  29. Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018. SemEval-2018 Task 11: Machine comprehension using commonsense knowledge. In Proceedings of the SemEval, pages 747–757, New Orleans, LA. find
  30. Soham Parikh, Ananya Sai, Preksha Nema, and Mitesh M Khapra. 2018. Eliminet: A model for eliminating options for reading comprehension with multiple choice questions. In Proceedings of the IJCAI-ECAI, pages 4272–4278, Stockholm, Sweden. find
  31. Anselmo Penas, Yusuke Miyao, Alvaro Rodrigo, Eduard H Hovy, and Noriko Kando. 2014. Overview of CLEF QA Entrance Exams Task 2014. In Proceedings of the CLEF, pages 1194–1200, Sheffield, UK. find
  32. David D Qian and Mary Schedl. 2004. Evaluation of an in-depth vocabulary knowledge measure for assessing reading performance. Language Testing, 21(1):28–52. find
  33. Sewon Min, Minjoon Seo, and Hannaneh Hajishirzi. 2017. Question answering through transfer learning from large fine-grained supervision data. In Proceedings of the ACL, pages 510–517, Vancouver, Canada. find
  34. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. In Preprint. find
  35. Paramita Mirza and Raffaella Bernardi. 2013. Ccg categories for distributional semantic models. In Proceedings of the RANLP, pages 467–474, Hissar, Bulgaria. find
  36. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the EMNLP, pages 2383–2392, Austin, TX.
  37. Kouider Mokhtari and Carla A Reichard. 2002. Assessing students’ metacognitive awareness of reading strategies. Journal of educational psychology, 94(2):249. find
  38. Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. CoQA: A conversational question answering challenge. Transactions of the Association of Computational Linguistics.
  39. Kouider Mokhtari and Ravi Sheorey. 2002. Measuring esl students’ awareness of reading strategies. Journal of developmental education, 25(3):2–11. find
  40. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the EMNLP, pages 193–203, Seattle, WA.
  41. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. In Proceedings of the NAACL-HLT, pages 839–849, San Diego, CA. find
  42. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, cs.CL/1611.09268v2.
  43. Hideyuki Shibuki, Kotaro Sakamoto, Yoshinobu Kano, Teruko Mitamura, Madoka Ishioroshi, Kelly Y Itakura, Di Wang, Tatsunori Mori, and Noriko Kando. 2014. Overview of the NTCIR-11 QA-Lab Task. In NTCIR. find
  44. Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In Proceedings of the EMNLP, pages 4208–4219, Brussels, Belgium.
  45. Saku Sugawara, Yusuke Kido, Hikaru Yokono, and Akiko Aizawa. 2017a. Evaluation metrics for machine reading comprehension: Prerequisite skills and readability. In Proceedings of the ACL, pages 806–817, Vancouver, Canada. find
  46. Saku Sugawara, Hikaru Yokono, and Akiko Aizawa. 2017b. Prerequisite skills for reading comprehension: Multi-perspective analysis of mctest datasets and systems. In Proceedings of the AAAI, pages 3089–3096, San Francisco, CA. find
  47. Min Tang, Jiaran Cai, and Hankz Hankui Zhuo. 2019. Multi-matching network for multiple choice reading comprehension. In Proceddings of the AAAI, Honolulu, HI. find
  48. Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Multi-range reasoning for machine comprehension. CoRR, cs.CL/1803.09074v1. find
  49. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the RepL4NLP, pages 191–200, Vancouver, Canada.
  50. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. QANet: Combining local convolution with global self-attention for reading comprehension. In Proceedings of the ICLR, Vancouver, Canada.
  51. Xingdi Yuan, Tong Wang, Caglar Gulcehre, Alessandro Sordoni, Philip Bachman, Saizheng Zhang, Sandeep Subramanian, and Adam Trischler. 2017. Machine comprehension by text-to-text neural question generation. In Proceedings of the RepL4NLP, pages 15–25, Vancouver, Canada. find
  52. Haichao Zhu, Furu Wei, Bing Qin, and Ting Liu. 2018. Hierarchical attention flow for multiple-choice reading comprehension. In Proceedings of the AAAI, pages 6077–6084, New Orleans, LA. find
  53. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE ICCV, pages 19–27, Santiago, Chile.
  54. Bo-Hsiang Tseng, Sheng-Syun Shen, Hung-Yi Lee, and Lin-Shan Lee. 2016. Towards machine comprehension of spoken content: Initial toefl listening comprehension test by machine. In Proceedings of the Interspeech, San Francisco, CA. find
  55. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the NIPS, pages 5998– 6008, Long Beach, CA.
  56. Liang Wang, Meng Sun, Wei Zhao, Kewei Shen, and Jingming Liu. 2018a. Yuanfudao at SemEval-2018 Task 11: Three-way attention and relational knowledge for commonsense machine comprehension. In Proceedings of the SemEval, pages 758–762, New Orleans, LA. find
  57. Shuohang Wang, Mo Yu, Shiyu Chang, and Jing Jiang. 2018b. A co-matching model for multi-choice reading comprehension. In Proceedings of the ACL, pages 1–6, Melbourne, Australia. find
  58. Georg Wiese, Dirk Weissenborn, and Mariana Neves. 2017. Neural domain adaptation for biomedical question answering. In Proceedings of the CoNLL, pages 281–289, Vancouver, Canada. find
  59. Yichong Xu, Jingjing Liu, Jianfeng Gao, Yelong Shen, and Xiaodong Liu. 2018. Dynamic fusion networks for machine reading comprehension. CoRR, cs.CL/1711.04964v2. find
  60. Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William Cohen. 2017. Semi-supervised QA with generative domain-adaptive nets. In Proceedings of the ACL, pages 1040–1050, Vancouver, Canada. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2019-improving-machine-reading-comprehension-general/