Elamur Research⌕
☰
home / papers / 2018 / WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference

WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference

Summary

QA systems for standardized science exams are getting better at picking correct answers, but they rarely explain why an answer is right. A major obstacle is the lack of gold explanations to train on. Current models also struggle to combine more than about two facts, because meaning drifts as facts are chained. To address this, the authors manually wrote explanations for nearly all publicly available elementary science exam questions (grades 3–5, about 1,680 items). Each explanation is stored both as free text and as an "explanation graph": a set of sentences linked to each other and to the question and answer through shared words. They also release a tablestore of semi-structured tables containing the knowledge used in these explanations. Their analysis suggests a typical explanation needs about six facts, more than an earlier estimate of four. Simulations of how knowledge grows and how much explanations overlap suggest the required knowledge base levels off at a manageable size. The resource offers direct supervision for multi-hop, explainable inference.

Key points

  • The WorldTree corpus contains explanations for 1,680 elementary science questions (3rd–5th grade), given both as free text and as lexically connected explanation graphs.
  • The accompanying tablestore has 62 semi-structured tables with 4,950 rows. It covers much of the knowledge needed for elementary science questions that are not spatial or mathematical.
  • Explanations average roughly 6 facts. An earlier estimate (Jansen et al., 2016) put this at 4. Inference methods typically fail beyond aggregating about 2 facts from free text.
  • In Monte-Carlo simulations, the number of questions whose explanations share 1, 2, 3 or more sentences grows logarithmically with corpus size.
  • The number of table rows needed to explain a given number of questions follows a proportional decay curve (R² = 0.99). It approaches a ceiling of about 6,000 rows at around 10,000 questions.

Interesting ideas

  • I1. Shared words as graph edges. Facts in an explanation are connected only when they share words, with no hand-labeled relations. This keeps annotation cheap while still making the multi-hop structure explicit, so it could work for building chained-reasoning datasets in other domains.
  • I2. Knowledge stored in tables designed for explanations. Facts live in typed, semi-structured table rows, so the same knowledge can be read as natural sentences or processed as structured data. This middle ground suits low-resource domains where full knowledge bases cost too much and text extraction is too noisy.
  • I3. Simulating corpus growth to estimate cost. Sampling question subsets many times and fitting a curve to the growth in required knowledge lets you predict the size of the full knowledge base before building it. This could help budget annotation for any new explanation or knowledge corpus.
  • I4. Measuring how much explanations are reused. Checking how many questions share explanation sentences shows how far knowledge transfers between questions. That tells you whether a retrieval-and-reuse approach to inference is realistic for a domain.
  • I5. Gold explanations instead of latent variables. The authors build explicit explanation supervision rather than treating explanation quality as a hidden variable learned from answer correctness. This sidesteps the drift problem for long reasoning chains and also gives a basis for evaluating explanations, not just answers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

65 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 62 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{jansen2018worldtree,
  title = {{WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference}},
  author = {Jansen, Peter and Wainwright, Elizabeth and Marmorstein, Steven and Morrison, Clayton},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  doi = {10.63317/28283yn8kzpn},
  url = {https://doi.org/10.63317/28283yn8kzpn}
}

References

24 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Berant, J., Chou, A., Frostig, R., and Liang, P. (2013). Semantic parsing on freebase from question-answer pairs. In EMNLP. find
  2. Clark, P., Harrison, P., and Balasubramanian, N. (2013). A study of the knowledge base requirements for passing an elementary science test. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, AKBC’13, pages 37–42. find
  3. Clark, P., Etzioni, O., Khot, T., Sabharwal, A., Tafjord, O., Turney, P. D., and Khashabi, D. (2016). Combining retrieval, statistics, and inference to answer elementary science questions. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Ariz… find
  4. Clark, P. (2015). Elementary school science and math tests as a driver for AI: take the aristo challenge! In Blai Bonet et al., editors, Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA., pages 4019–4021. AAAI Press. find
  5. Dalvi, B., Bhakthavatsalam, S., Clark, C., Clark, P., Etzioni, O., Fader, A., and Groeneveld, D. (2016). IKE - an interactive tool for knowledge extraction. In Proceedings of the 5th Workshop on Automated Knowledge Base Construction, AKBC@NAACL-HLT 2016, San Diego, CA, USA, June 17, 2016, pages 12–1… find
  6. Etzioni, O., Fader, A., Christensen, J., Soderland, S., and Mausam, M. (2011). Open information extraction: The second generation. In Proceedings of the 20th International Joint Conference on Artificial Intelligence (ICJAI), pages 3–10. find
  7. Fried, D., Jansen, P., Hahn-Powell, G., Surdeanu, M., and Clark, P. (2015). Higher-order lexical semantic models for non-factoid answer reranking. Transactions of the Association for Computational Linguistics, 3:197–210. find
  8. Jansen, P., Balasubramanian, N., Surdeanu, M., and Clark, P. (2016). What’s in an explanation? characterizing knowledge and inference requirements for elementary science exams. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2956… find
  9. The COLING 2016 Organizing Committee. find
  10. Jansen, P., Sharp, R., Surdeanu, M., and Clark, P. (2017). Framing qa as building and ranking intersentence answer justifications. Computational Linguistics. find
  11. Jansen, P. (2017). A study of automatically acquiring explanatory inference patterns from corpora of explanations: Lessons from elementary science exams. In Proceedings of the 2017 Workshop on Automated Knowledge Base Construction, AKBC’17. find
  12. Jauhar, S. K., Turney, P. D., and Hovy, E. H. (2016). Tables as semi-structured knowledge for question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL). find
  13. Khashabi, D., Khot, T., Sabharwal, A., Clark, P., Etzioni, O., and Roth, D. (2016). Question answering via integer programming over semi-structured knowledge. In Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI’16, pages 1145–1152. find
  14. Khot, T., Balasubramanian, N., Gribkoff, E., Sabharwal, A., Clark, P., and Etzioni, O. (2015). Exploring markov logic networks for question answering. In EMNLP. find
  15. Khot, T., Sabharwal, A., and Clark, P. (2017). Answering complex questions using open information extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 2: Short Papers, pages 311–316. find
  16. Legare, C. H. (2014). The contributions of explanation and exploration to children’s scientific reasoning. Child Development Perspectives, 8(2):101–106. find
  17. Pasupat, P. and Liang, P. (2015). Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL). find
  18. Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). ”why should i trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 1135–1144, New York, NY, USA. ACM. find our card
  19. Rittle-Johnson, B. and Loehr, A. M. (2016). Eliciting explanations: Constraints on when self-explanation aids learning. Psychonomic bulletin & review, pages 1–10. find
  20. Roscoe, R. D. and Chi, M. T. (2007). Understanding tutor learning: Knowledge-building and knowledge-telling in peer tutors’ explanations and questions. Review of Educational Research, 77(4):534–574. find
  21. Schoenick, C., Clark, P., Tafjord, O., Turney, P., and Etzioni, O. (2017). Moving beyond the turing test with the allen ai science challenge. Communications of the ACM, 60(9):60–64. find
  22. Sharp, R., Surdeanu, M., Jansen, P., Valenzuela-Escárcega, M. A., Clark, P., and Hammond, M. (2017). Tell me why: Using question answering as distant supervision for answer justification. In Proceedings of the Conference on Computational Natural Language Learning (CoNLL). find
  23. Sun, H., Ma, H., He, X., Yih, W.-t., Su, Y., and Yan, X. (2016). Table cell search for question answering. In Proceedings of the 25th International Conference on World Wide Web (WWW), pages 771–782. find
  24. Yin, P., Duan, N., Kao, B., Bao, J., and Zhou, M. (2015). Answering questions with complex semantic constraints on open knowledge bases. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 1301–1310. ACM. find

↑ back to top

This page in Markdown →