Elamur Research⌕
☰
home / papers / 2016 / Moving Beyond the Turing Test with the Allen AI Science Challenge

Moving Beyond the Turing Test with the Allen AI Science Challenge

Paper · 2016 · Carissa Schoenick, Peter Clark, Oyvind Tafjord et al.

Summary

The authors argue that the Turing Test no longer works as a benchmark for AI. Its rules are loosely specified, chatbots can game it by fooling judges, and it yields only a pass/fail verdict instead of a graded score. As a measurable alternative, AI2 ran the Allen AI Science Challenge on Kaggle from October 2015 to February 2016. Systems had to answer four-option multiple-choice 8th grade science questions taken from licensed exam material. To discourage manual answering, the genuine questions were mixed with many machine-generated decoys. All three winning entries depended mostly on information retrieval over carefully curated corpora. They fed signals such as word2vec/GloVe similarity, PMI and features of the question's form into gradient boosting or logistic regression models. The top score was just under 60%. That clearly beats a Lucene-over-Wikipedia baseline but is far from mastering the exam. The paper concludes that retrieval-based methods have hit a ceiling. Further progress will require deeper semantic reasoning and commonsense knowledge, which is the direction of AI2's Project Aristo.

Key points

  • Participation: 780 teams took part and 170 submitted final models. The top three received $50,000, $20,000 and $10,000, and winners had to open-source their code.
  • Data: 5,083 questions were licensed in total, with 2,500 used for training.
    • The validation set had 8,132 items, of which only 800 were real.
    • The test set had 21,298 items, of which 2,583 were real.
    • AI2 later published some subsets on its website.
  • Scores: Random guessing gives 25%, and the Lucene/Wikipedia baseline reached 40.7% on the test set. The winners scored 59.31% (Chaim Linhart, "Cardal"), 58.34% (Talkwalker, "poweredByTalkwalker") and 58.26% (Alejandro Mosquera), a spread of only 1.05%.
  • Winning methods:
    • Linhart averaged 15 small gradient boosting models that scored each option separately; by his estimate, IR features alone gave about 55%.
    • Talkwalker built a 180GB Lucene index and used XGBoost pairwise ranking plus hashed term–definition features.
    • Mosquera classified the 12 ordered answer pairs into three classes with logistic regression over Elastic Search features.
  • Open problem: Questions that every winner missed, such as the earthquakes/plate tectonics and star-brightness examples, require chained reasoning. Existing QA systems do not handle this kind of reasoning, including those from TREC, MUC and FreeBase-based work, as well as Wolfram Alpha, Evi and START.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States353 works · 200,043 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States275 works · 97,634 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Artificial intelligence · 49 works

Cite

Cite as Carissa Schoenick, Peter Clark, Oyvind Tafjord, Peter Turney and Oren Etzioni. 2016. Moving Beyond the Turing Test with the Allen AI Science Challenge. arXiv:1604.04315.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1604.04315 · arXiv 1604.04315

@misc{schoenick2016moving,
  title = {{Moving Beyond the Turing Test with the Allen AI Science Challenge}},
  author = {Carissa Schoenick, Peter Clark, Oyvind Tafjord, Peter Turney and Oren Etzioni},
  year = {2016},
  doi = {10.48550/arxiv.1604.04315},
  eprint = {1604.04315},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1604.04315}
}

References

17 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan M. Turing. Computing machinery and intelligence. Mind, LIX(236):433–460, October 1950. find
  2. BBC. Computer AI passes Turing test in 'world first' Turing Test. BBC News. 9 June 2014. http://www.bbc.com/news/technology-27762088 ↗
  3. Aron, J. Software tricks people into thinking it is human. New Scientist (Issue 2829), Sept 2011. find
  4. Marcus, G., Rossi, F., Veloso, M. (Eds), Beyond the Turing Test (AI Magazine Special Edition), AI Magazine, 37 (1), Spring 2016. find
  5. Turk, V. The Plan to Replace the Turing Test with a ‘Turing Olympics’. Motherboard. 28 January 2015. http://motherboard.vice.com/read/the-plan-to-replace-the-turing-test-with-a-turing-olympics ↗
  6. Clark, P., Etzioni, O. My Computer is an Honor Student - But how Intelligent is it? Standardized Tests as a Measure of AI. In AI Magazine 37 (1), Spring 2016. find
  7. "The Allen AI Science Challenge." Kaggle, 13 Feb. 2016, https://www.kaggle.com/c/the-allen-ai-science-challenge. ↗
  8. "Data - Allen Institute for Artificial Intelligence." Allen Institute for Artificial Intelligence, http://allenai.org/data. Accessed 20 Oct. 2016. ↗
  9. Grishman, R., Sundheim, B. Message Understanding Conference-6: A Brief History. In COLING (Vol. 96, pp. 466-471), 1996. — Message Understanding Conference- 6: A Brief History
  10. Voorhees, E., Ellis, A. (Eds)Proc. 24th Text REtrieval Conference (TREC 2015), Publication SP 500-319, NIST (http://trec.nist.gov/ ), 2015. find
  11. Yao, X., Van Durme, B. Information Extraction over Structured Data: Question Answering with Freebase. In ACL (1) (pp. 956-966), 2014. find
  12. Berant, J., Chou, A., Frostig, R., Liang, P. Semantic Parsing on Freebase from Question-Answer Pairs. In EMNLP (Vol. 2, No. 5, p. 6), 2013.
  13. Fader, A., Zettlemoyer, L., & Etzioni, O. Open question answering over curated and extracted knowledge bases. In Proc 20th ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining (pp. 1156-1165). ACM, 2014. find
  14. Wolfram, S. Making the World's Data Computable. Proc. Wolfram Data Summit, 2010. (http://blog.stephenwolfram.com/2010/09/making-the-worlds-data-computable/ ) find
  15. Simmons, J. True Knowledge: The Natural Language Question Answering Wikipedia for Facts. In: Semantic Focus, Feb 2008. find
  16. Katz, B., Borchardt, G., Felshin, S. Natural Language Annotations for Question Answering. Proc 19th Int FLAIRS Conference (FLAIRS 2006), 2006. (http://start.csail.mit.edu) find
  17. Sachan, M., Dubey, A., Xing, E. Science Question Answering using Instructional Materials. arXiv preprint at arXiv:1602.04375 http://arxiv.org/pdf/1602.04375.pdf ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schoenick-2016-moving-turing-test-allen/