Elamur Research
☰
home / papers / 2017 / Learning to Ask: Neural Question Generation for Reading Comprehension

Learning to Ask: Neural Question Generation for Reading Comprehension

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1705.00106

Summary

The paper tackles automatic question generation (QG) for reading comprehension: given a sentence from a passage, the system produces a natural question about its content. Earlier systems relied on handcrafted declarative-to-question transformation rules, often followed by a feature-based ranker. Their questions tend to copy the source wording and are therefore easy to answer. The authors instead treat QG as end-to-end sequence-to-sequence learning. They use an LSTM encoder-decoder with global attention, trained on sentence-question pairs from SQuAD and requiring no rules or NLP pipeline. A second variant also encodes the surrounding paragraph, and they test pre-trained versus learned word embeddings. On automatic metrics such as BLEU and METEOR, the neural model clearly beats an IR baseline, a phrase-based SMT system and the rule-based Heilman & Smith (H&S) system. Human judges also found its questions more grammatical, fluent and harder to answer. The work matters because it shows that data-driven abstractive generation can paraphrase and compress source text. That makes it useful for building educational materials and for creating QA datasets.

Key points

  • Architecture: a bidirectional LSTM encodes the sentence, and an LSTM decoder with Luong-style global attention (bilinear scoring) generates the question. The paragraph-aware variant uses a "Y"-shaped design: a second BiLSTM encodes the paragraph, and both encodings are concatenated to initialize the decoder.
  • Baselines: the system is compared on SQuAD against IR, an SMT system (Moses), DirectIn (which outputs the input sentence directly) and H&S. DirectIn and H&S score similarly because both stay very close to the source wording.
  • Embeddings and paragraph context: pre-trained word embeddings generally improve results. Adding paragraph context slightly lowers overall automatic scores, which the authors attribute to noise in the paragraph.
  • Human evaluation: the system outperforms H&S on every criterion. It was ranked best in 38.4% of judgments, with an average rank of 1.94 and Krippendorff's Alpha of 0.236.
  • Dev-set analysis: in a sample of 346 dev pairs, about 70% of questions need only the sentence and about 20% need wider paragraph context. Roughly 9.5% were judged not askable. The paragraph-level model performs best on the questions that need paragraph context.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

225 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 217 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1705.00106 · arXiv 1705.00106

@misc{du2017learning,
  title = {{Learning to Ask: Neural Question Generation for Reading Comprehension}},
  author = {Du, Xinya and Shao, Junru and Cardie, Claire},
  year = {2017},
  doi = {10.48550/arxiv.1705.00106},
  eprint = {1705.00106},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1705.00106}
}

References

39 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations Workshop (ICLR). find our card
  2. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin,… find our card
  3. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 . find our card
  4. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langua… find
  5. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Com… find our card
  6. Kenneth Mark Colby, Sylvia Weber, and Franklin Dennis Hilf. 1971. Artificial paranoia. Artificial Intelligence 2(1):1–25. https://doi.org/10.1016/0004-3702(71)90002-6. — [doi:10.1016/0004-3702(71)90002-6](https://doi.org/10.1016/0004-3702%2871%2990002-6) ↗
  7. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation. Association for Computational Linguistics, Baltimore, Maryland, USA, pages 376– 380. http://www.aclweb.… ↗ our card
  8. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified kneser-ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Sofi… find our card
  9. Michael Heilman and Noah A. Smith. 2010. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Los Angeles, Cali… find
  10. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS). pages 1693–1701. find our card
  11. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780. find
  12. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, G… find
  13. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. ArXiv e-prints . find our card
  14. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… find
  15. Igor Labutov, Sumit Basu, and Lucy Vanderwende. 2015. Deep questions without deep understanding. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association f… find
  16. Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions and reversals. In Soviet physics doklady. volume 10, page 707. find
  17. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Stan Szpakowicz Marie-Francine Moens, editor, Text Summarization Branches Out: Proceedings of the ACL-04 Workshop. Association for Computational Linguistics, Barcelona, Spain, pages 74–81. http://aclweb.org/anthology/W/W0… ↗
  18. David Lindberg, Fred Popowich, John Nesbit, and Phil Winne. 2013. Generating natural language questions to support learning on-line. In Proceedings of the 14th European Workshop on Natural Language Generation. Association for Computational Linguistics, Sofia, Bulgaria, pages 105–114. http://www.aclw… ↗
  19. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 1412– 1421. htt… find
  20. Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the rare word problem in neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language… find our card
  21. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny F., Steven B., and David M. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics,… find our card
  22. Karen Mazidi and Rodney D. Nielsen. 2014. Linguistic considerations in automatic question generation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Baltimore, Maryland, pages 321–326. ht… find
  23. Ruslan Mitkov and Le An Ha. 2003. Computer-aided generation of multiple-choice tests. In Jill Burstein and Claudia Leacock, editors, Proceedings of the HLT-NAACL 03 Workshop on Building Educational Applications Using Natural Language Processing. pages 17–22. http://www.aclweb.org/anthology/W03-0203.… ↗
  24. Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. 2016. Generating natural questions about an image. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational L… find
  25. Jack Mostow and Wei Chen. 2009. Generating instruction automatically for the reading strategy of self-questioning. In Proceedings of the 2nd Workshop on Question Generation (AIED 2009). pages 465–472. find
  26. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 . find our card
  27. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pages… find our card
  28. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pages 1532–1543. http://www.acl… ↗
  29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Austin, Texas, pages 238… find our card
  30. Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washingto… find our card
  31. Stephen E. Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Springer-Verlag New York, Inc., Ne… find
  32. Vasile Rus, Brendan Wyse, Paul Piwek, Mihai Lintean, Svetlana Stoyanchev, and Cristian Moldovan. 2010. The first question generation shared task evaluation challenge. In Proceedings of the 6th International Natural Language Generation Conference. Association for Computational Linguistics, Stroudsbur… find
  33. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 379–389. http://aclw… find
  34. Iulian Vlad Serban, Alberto García-Durán, Caglar Gulcehre, Sungjin Ahn, Sarath Chandar, Aaron Courville, and Yoshua Bengio. 2016. Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus. In Proceedings of the 54th Annual Meeting of the Association for Comp… find
  35. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems (NIPS). pages 3104–3112. find
  36. Lucy Vanderwende. 2008. The importance of being important: Question generation. In Proceedings of the 1st Workshop on the Question Generation Shared Task Evaluation Challenge, Arlington, VA. find
  37. Joseph Weizenbaum. 1966. Eliza—a computer program for the study of natural language communication between man and machine. Commun. ACM 9(1):36–45. https://doi.org/10.1145/365153.365168. — [doi:10.1145/365153.365168](https://doi.org/10.1145/365153.365168) ↗
  38. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. In International Conference on Learning Representations Workshop (ICLR). find our card
  39. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In ICML. volume 14, pages 77–81. find our card