Elamur Research
☰
home / papers / 2017 / Crowdsourcing Multiple Choice Science Questions

Crowdsourcing Multiple Choice Science Questions

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W17-4413.pdf

Summary

Answering science exam questions requires background knowledge and reasoning, but there is little in-domain training data for it. Earlier collections were too small to train neural models. Crowd workers also lack domain expertise, so they tend to write repetitive or off-topic questions with weak wrong options. The authors propose a crowdsourcing pipeline that turns question writing into guided selection and editing. First, a noisy classifier picks promising paragraphs from science study texts, and a worker chooses one and writes a question with its answer. Second, a random-forest ranker trained on real exam questions suggests candidate distractors, and another worker uses these suggestions to write three wrong options. The result is SciQ: 13,679 four-option science questions, most of them linked to their source passage. Human judges could not tell SciQ items apart from genuine exam questions. Adding SciQ to the training data also raised the accuracy of neural readers on real 4th- and 8th-grade exams. SciQ is therefore useful both as a medium-sized benchmark and as extra in-domain training data.

Key points

  • SciQ cost $10,415 in total and comes in two versions: a multiple-choice version and a direct-answer version, where the system extracts the answer span from a passage. It is distributed via allenai.org/data.
  • The distractor ranker is learned from real exam questions instead of relying only on similarity. It still uses several classic similarity heuristics (for example WordNet-based or distributional measures) as features.
  • On the multiple-choice version, the AS Reader and GA Reader did not beat a plain Lucene IR baseline, even with the same background corpus. TableILP did poorly because its hand-built knowledge tables do not cover the SciQ topics. Human accuracy was estimated from 650 questions answered by 13 people.
  • On the direct-answer version, BiDAF scored 66.7% exact match and 75.7 F1, only 1.3 and 1.6 points below its SQuAD results.
  • The real-exam training sets were about 3,200 8th-grade and 1,200 4th-grade questions. Adding SciQ improved both readers at both grade levels, while the authors' earlier purely synthetic data had caused overfitting and lower scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

324 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 309 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-4413 · arXiv 1707.06209

@inproceedings{welbl2017crowdsourcing,
  title = {{Crowdsourcing Multiple Choice Science Questions}},
  author = {Welbl, Johannes and Liu, Nelson F. and Gardner, Matt},
  booktitle = {Proceedings of the 3rd Workshop on Noisy User-generated Text},
  year = {2017},
  doi = {10.18653/v1/w17-4413},
  eprint = {1707.06209},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w17-4413}
}

References

36 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Manish Agarwal and Prashanth Mannem. 2011. Automatic gap-fill question generation from text books. In Proceedings of the 6th Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, Stroudsburg, PA, USA, IUNLPBEA ’11, pages 56–64. http://dl.… ↗
  2. Itziar Aldabe and Montse Maritxalar. 2010. Automatic Distractor Generation for Domain Specific Texts, Springer Berlin Heidelberg, Berlin, Heidelberg, pages 27–38. find
  3. Rui Correia, Jorge Baptista, Nuno Mamede, Isabel Trancoso, and Maxine Eskenazi. 2010. Automatic generation of cloze question distractors. In Proceedings of the Interspeech 2010 Satellite Workshop on Second Language Studies: Acquisition, Learning, Education and Technology, Waseda University, Tokyo, J… find
  4. Bhuwan Dhingra, Hanxiao Liu, William W. Cohen, and Ruslan Salakhutdinov. 2016. Gated-attention readers for text comprehension. CoRR abs/1606.01549. http://arxiv.org/abs/1606.01549. ↗
  5. Michael Heilman and Noah A. Smith. 2010. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Stroudsburg, PA,… find
  6. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeti… find
  7. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS). http://arxiv.org/abs/1506.03340. ↗ our card
  8. Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. CoRR abs/1506.02075. http://arxiv.org/abs/1506.02075. ↗
  9. Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. 2016. Wikireading: A novel large-scale language understanding task over wikipedia. CoRR abs/1608.03542. http://arxiv.org/abs/1608.03542. ↗
  10. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. find
  11. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. CoRR abs/1511.02301. http://arxiv.org/abs/1511.02301. ↗ our card
  12. Leo Breiman. 2001. Random forests. Machine Learning 45(1):5–32. find
  13. Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. CoRR abs/1603.01547. http://arxiv.org/abs/1603.01547. ↗
  14. Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR abs/1611.09268. http://arxiv.org/abs/1611.09268. ↗ our card
  15. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David A. McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016. pages 2230–22… find
  16. Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni, and Dan Roth. 2016. Question answering via integer programming over semi-structured knowledge. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15… find
  17. Andreas Papasalouros, Konstantinos Kanaris, and Konstantinos Kotis. 2008. Automatic generation of multiple choice questions from domain ontologies. In Miguel Baptista Nunes and Maggie McPherson, editors, e-Learning. IADIS, pages 427–434. find
  18. Tushar Khot, Niranjan Balasubramanian, Eric Gribkoff, Ashish Sabharwal, Peter Clark, and Oren Etzioni. 2015. Exploring markov logic networks for question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-2… find
  19. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031 . find our card
  20. Yang Li and Peter Clark. 2015. Answering elementary science questions by constructing coherent scenes using background knowledge. In EMNLP. pages 2007–2012. find
  21. Mausam, Michael Schmitz, Robert Bart, Stephen Soderland, and Oren Etzioni. 2012. Open language learning for information extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computation… find
  22. Ruslan Mitkov and Le An Ha. 2003. Computer-aided generation of multiple-choice tests. In Proceedings of the HLT-NAACL 03 Workshop on Building Educational Applications Using Natural Language Processing - Volume 2. Association for Computational Linguistics, Stroudsburg, PA, USA, HLT-NAACL-EDUC ’03, pa… find
  23. Ruslan Mitkov, Le An Ha, Andrea Varga, and Luz Rello. 2009. Semantic similarity of distractors in multiple-choice tests: Extrinsic evaluation. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics. Association for Computational Linguistics, Stroudsburg, PA, USA, GEMS ’09… find
  24. Jack Mostow and Hyeju Jang. 2012. Generating diagnostic multiple choice comprehension cloze questions. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP. Association for Computational Linguistics, Stroudsburg, PA, USA, pages 136–146. http://dl.acm.org/citation.cfm… ↗
  25. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP). pages 1532– 1543. http://www.aclwe… ↗
  26. Juan Pino and Maxine Esknazi. 2009. Semi-automatic generation of cloze question distractors effect of students’ l1. In SLaTE. ISCA, pages 65–68. find
  27. Juan Pino, Michael Heilman, and Maxine Eskenazi. 2008. A Selection Strategy to Improve Cloze Question Quality. In Proceedings of the Workshop on Intelligent Tutoring Systems for Ill-Defined Domains. 9th International Conference on Intelligent Tutoring Systems.. find
  28. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  29. Mrinmaya Sachan, Avinava Dubey, and Eric P. Xing. 2016. Science question answering using instructional materials. CoRR abs/1602.04375. http://arxiv.org/abs/1602.04375. ↗
  30. Keisuke Sakaguchi, Yuki Arase, and Mamoru Komachi. 2013. Discriminative approach to fill-in-the-blank quiz generation for language learners. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 2: Short Papers.… find
  31. Carissa Schoenick, Peter Clark, Oyvind Tafjord, Peter Turney, and Oren Etzioni. 2016. Moving beyond the turing test with the allen ai science challenge. arXiv preprint arXiv:1604.04315 . find
  32. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. CoRR abs/1611.01603. http://arxiv.org/abs/1611.01603. ↗ our card
  33. Alessandro Sordoni, Phillip Bachman, and Yoshua Bengio. 2016. Iterative alternating neural attention for machine reading. CoRR abs/1606.02245. http://arxiv.org/abs/1606.02245. — [arXiv:1606.02245](https://arxiv.org/abs/1606.02245) ↗
  34. Eiichiro Sumita, Fumiaki Sugaya, and Seiichi Yamamoto. 2005. Measuring non-native speakers’ proficiency of english by using a test with automatically-generated fill-in-the-blank questions. In Proceedings of the Second Workshop on Building Educational Applications Using NLP. Association for Computati… find
  35. Yi Yang, Scott Wen-tau Yih, and Chris Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. ACL Association for Computational Linguistics. https://www.microsoft.com/en-us/research/publication/wikiqa-a-challenge-datasetfor-open-domain-question-answering/. ↗ our card
  36. Torsten Zesch and Oren Melamud. 2014. Automatic generation of challenging distractors using context-sensitive inference rules. In Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications, BEA@ACL 2014, June 26, 2014, Baltimore, Maryland, USA. pages 143– 148. h… find