Elamur Research⌕
☰
home / papers / 2018 / Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Paper · 2018 · Peter Clark, Isaac Cowhey, Oren Etzioni et al.

Summary

The authors from the Allen Institute for AI argue that popular QA datasets (SQuAD, SNLI, TriviaQA) mostly test the retrieval of answers based on surface-level matches in the text. Success on them therefore does not mean that models can actually reason. To shift research toward harder tasks, they created the AI2 Reasoning Challenge (ARC). It consists of 7787 natural, grade-school, multiple-choice science questions taken from real standardized tests. The questions are divided into two parts. The Challenge Set contains those answered incorrectly by both a retrieval-based IR solver and a statistical PMI solver that counts word co-occurrence. The rest form the Easy Set. Along with the questions, the authors released the ARC Corpus of science sentences and neural models adapted for answer selection. The main result: none of the tested approaches, including the best models for SQuAD and SNLI, noticeably outperformed random guessing on the Challenge Set. On the Easy Set, the same systems score roughly 55–65%. As a result, ARC has become an open benchmark for testing genuine reasoning and knowledge use in QA.

Key points

  • Data size: 7787 questions for grades 3–9, usually with four answer options, without diagrams. The Challenge Set contains 2590 questions, the Easy Set 5197. The train/dev/test split is 3370/869/3548. Approximately 43% of the dataset consists of the public questions from the 2016 Kaggle competition.
  • ARC Corpus: about 14M science-related sentences. According to a sample check, the corpus contains the necessary knowledge for approximately 95% of the Challenge Set questions. Its use is optional.
  • Baseline models: IR, PMI, TupleInference, TableILP, as well as the neural models DecompAttn, DGEM, DGEM-OpenIE and BiDAF, adapted to multiple-choice selection. The code is published in the allenai/arc-solvers repository.
  • Results on the Challenge Set: the best result was achieved by DGEM at 27.11%. Random guessing yields 25.02%, with a confidence interval of ±2.5%. Thus, the advantage over the random baseline is not significant.
  • Reason for failure: almost all systems first select sentences using simple IR retrieval. Such retrieval finds sentences similar to the question rather than a set of facts that need to be chained together to arrive at the answer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksQuestion answering · 441 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Question answering · 53 works

Cite

Cite as Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1803.05457 · arXiv 1803.05457

@misc{clark2018think,
  title = {{Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge}},
  author = {Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick and Oyvind Tafjord},
  year = {2018},
  doi = {10.48550/arxiv.1803.05457},
  eprint = {1803.05457},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1803.05457}
}

References

32 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Brachman. Selected grand challenges in cognitive science. Technical Report Tech Report 05-1218, MITRE, 2005. find
  2. K. W. Church and P. Hanks. Word association norms, mutual information and lexicography. In 27th ACL, pp. 76–83, 1989. — Word Association Norms, Mutual Information, and Lexicography
  3. P. Clark and O. Etzioni. My Computer is an Honor Student but how Intelligent is it? Standardized Tests as a Measure of AI. AI Magazine, 2016. (To appear). find
  4. P. Clark, O. Etzioni, T. Khot, A. Sabharwal, O. Tafjord, P. D. Turney, and D. Khashabi. Combining retrieval, statistics, and inference to answer elementary science questions. In AAAI, pp. 2580–2586, 2016. find
  5. E. Davis. How to write science questions that are easy for people and hard for computers. AI magazine, 37(1):13– 22, 2016. find
  6. A. Fujita, A. Kameda, A. Kawazoe, and Y. Miyao. Overview of todai robot project and evaluation framework of its nlp-based problem solving. In Proc. LREC’14, vol. 36, page 36, 2014. find
  7. S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. Bowman, and N. A. Smith. Annotation artifacts in natural language inference data. In Proc. of NAACL, 2018.
  8. M. Henaff, J. Weston, A. Szlam, A. Bordes, and Y. LeCun. Tracking the world state with recurrent entity networks. CoRR, abs/1612.03969, 2016. find
  9. K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pp. 1693–1701, 2015.
  10. R. Jia and P. Liang. Adversarial examples for evaluating reading comprehension systems. In Proc. EMNLP’17, 2017.
  11. M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proc. ACL’17, July 2017.
  12. A. Kembhavi, M. Seo, D. Schwenk, J. Choi, A. Farhadi, and H. Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In CVPR, 2017. find
  13. D. Khashabi, T. Khot, A. Sabharwal, P. Clark, O. Etzioni, and D. Roth. Question answering via integer programming over semi-structured knowledge. In IJCAI, 2016. find
  14. D. Khashabi, T. Khot, A. Sabharwal, and D. Roth. Question answering as global reasoning over semantic abstractions. In AAAI’18, 2018a. find
  15. D. Khashabi, T. K. A. Sabharwal, and D. Roth. Question answering as global reasoning over semantic abstractions. In AAAI, 2018b. find
  16. T. Khot, A. Sabharwal, and P. Clark. Answering complex questions using open information extraction. In ACL, 2017.
  17. T. Khot, A. Sabharwal, and P. Clark. SciTail: A textual entailment dataset from science question answering. In AAAI, 2018. find
  18. NII. Evaluation of information access technologies. In Proc. 13th NTCIR Conf, 2017. find
  19. A. P. Parikh, O. Täckström, D. Das, and J. Uszkoreit. A decomposable attention model for natural language inference. In EMNLP, 2016.
  20. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text. In Proc. EMNLP’16, 2016.
  21. M. Richardson. Mctest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP’13, 2013.
  22. C. Schoenick, P. Clark, O. Tafjord, P. Turney, and O. Etzioni. Moving beyond the turing test with the allen ai science challenge. Communications of the ACM, 60(9): 60–64, 2017.
  23. M. Seo, S. Min, A. Farhadi, and H. Hajishirzi. Query-reduction networks for question answering. In ICLR, 2017a. find
  24. M. J. Seo, H. Hajishirzi, A. Farhadi, and O. Etzioni. Diagram understanding in geometry questions. In AAAI, pp. 2831–2838, 2014. find
  25. M. J. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. Bidirectional attention flow for machine comprehension. In Proc. ICLR’17, 2017b.
  26. T. Simonite. Ai beat humans at reading! maybe not. In WIRED Magazine, 2018. find
  27. E. Strickland. Can an ai get into the university of tokyo? IEEE Spectrum, August 2013. find
  28. A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830, 2016.
  29. J. Welbl, N. F. Liu, and M. Gardner. Crowdsourcing multiple choice science questions. In Workshop on Noisy User-generated Text, 2017a.
  30. J. Welbl, P. Stenetorp, and S. Riedel. Constructing datasets for multi-hop reading comprehension across documents. arXiv preprint arXiv:1710.06481, 2017b.
  31. J. Weston, A. Bordes, S. Chopra, A. M. Rush, B. van Merriënboer, A. Joulin, and T. Mikolov. Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015.
  32. J. Weston, S. Chopra, and A. Bordes. Memory networks. arXiv preprint arXiv:1410.3916, 2014. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/