Elamur Research⌕
☰
home / papers / 2017 / Answering Complex Questions Using Open Information Extraction

Answering Complex Questions Using Open Information Extraction

Paper · 2017 · Tushar Khot, Ashish Sabharwal and Peter Clark

Summary

The paper tackles multiple-choice elementary and middle-school science questions that need several facts combined. Earlier Open IE-based QA could only handle single-fact lookups, and stronger structured reasoners such as TableILP depended on hand-built tables and join rules. The authors present TupleInf, a solver that reasons directly over automatically extracted (subject; predicate; objects) tuples. To build the knowledge, they retrieve domain-relevant sentences from a 280 GB web-derived corpus with ElasticSearch and run Open IE v4 on them. For each question, they pick tuples from this KB using TF-IDF and also extract extra tuples from text on the fly. An Integer Linear Programming model then finds the best support graph linking question terms, tuple fields and one answer choice. Compared with TableILP, the formulation is simpler and better suited to noisy tuples. It weights question terms by position and rarity, requires the tuple subject to be used, and enforces word order. TupleInf beats TableILP by more than 11 points without any curated knowledge. It also adds value in ensembles with IR-based solvers. This shows that structured QA can work without manually built resources.

Key points

  • On the 4th Grade set (1,304 test questions), TupleInf scores 51.7% versus 39.9% for TableILP. On the 8th Grade set (282 test questions), it scores 51.6% versus 34.1%.
  • When both solvers receive identical knowledge (curated tables plus tuples), TupleInf still leads: 47.5% vs. 42.1% on 4th grade and 48.0% vs. 37.9% on 8th grade. On the Regents set, it roughly matches TableILP (61.4% vs. 61.5%) without specialized rules.
  • The ILP is roughly half the size of TableILP's (about 1,628 vs. 3,403 constraints). Overall answering time drops by 68.6%.
  • TupleInf and the IR solver disagree on 47.3% of training questions, so they complement each other. IR + TupleInf reaches 55.3% (4th grade), and adding the PMI solver gives 58.2%, compared with 56.7% for TableILP's ensemble.
  • The error analysis names four failure types: lossy Open IE extraction, bad phrase alignment, missed salient words, and questions that are out of the solver's scope. The model only combines parallel evidence, and multi-hop chaining is left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States219 works · 89,712 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

🧭 Topics 1Research subjects and languages the work is about.Information extraction · 109 works

Cite

Cite as Tushar Khot, Ashish Sabharwal and Peter Clark. 2017. Answering Complex Questions Using Open Information Extraction. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-2049

@inproceedings{khot2017answering,
  title = {{Answering Complex Questions Using Open Information Extraction}},
  author = {Tushar Khot, Ashish Sabharwal and Peter Clark},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-2049},
  url = {https://doi.org/10.18653/v1/p17-2049}
}

References

22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke S. Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In EMNLP.
  2. Tobias Achterberg. 2009. SCIP: solving constraint integer programs. Math. Prog. Computation 1(1):1– 41. find
  3. George Miller. 1995. WordNet: a lexical database for english. Communications of the ACM 38(11):39– 41.
  4. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In IJ-CAI.
  5. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In ACL.
  6. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Empirical Methods in Natural Language Processing (EMNLP).
  7. Jonathan Berant and Percy Liang. 2014. parsing via paraphrasing. In ACL. find
  8. Matthew Richardson and Pedro Domingos. 2006. Markov logic networks. Machine learning 62(1– 2):107–136. find
  9. Semantic Arpit Sharma, Nguyen Ha Vo, Somak Aditya, and Chitta Baral. 2015. Towards addressing the winograd schema challenge - building and using a semantic parser and a knowledge hunting module. In IJ-CAI. find
  10. Eric Brill, Susan Dumais, and Michele Banko. 2002. An analysis of the AskMSR question-answering system. In Proceedings of EMNLP. pages 257–264. find
  11. Gong Cheng, Weixi Zhu, Ziwei Wang, Jianghui Chen, and Yuzhong Qu. 2016. Taking up the Gaokao Challenge: An information retrieval approach. In IJCAI. find
  12. Huan Sun, Hao Ma, Xiaodong He, Wen tau Yih, Yu Su, and Xifeng Yan. 2016. Table cell search for question answering. In WWW. find
  13. Peter Clark. 2015. Elementary school science and math tests as a driver for AI: take the Aristo challenge! In 29th AAAI/IAAI. Austin, TX, pages 4019–4021. find
  14. Pengcheng Yin, Nan Duan, Ben Kao, Jun-Wei Bao, and Ming Zhou. 2015. Answering questions with complex semantic constraints on open knowledge bases. In CIKM. find
  15. Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Turney, and Daniel Khashabi. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In 30th AAAI. find
  16. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010. Recognizing textual entailment: Rational, evaluation and approaches–erratum. Natural Language Engineering 16(01):105–105. find
  17. Anthony Fader, Luke S. Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question answering. In ACL.
  18. Anthony Fader, Luke S. Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In KDD. find
  19. David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, et al. 2010. Building Watson: An overview of the DeepQA project. AI Magazine 31(3):59–79. find
  20. David Howell. 2012. Statistical methods for psychology. Cengage Learning. find
  21. Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni, and Dan Roth. 2016. Question answering via integer programming over semi-structured knowledge. In IJCAI. find
  22. Tushar Khot, Niranjan Balasubramanian, Eric Gribkoff, Ashish Sabharwal, Peter Clark, and Oren Etzioni. 2015. Exploring Markov logic networks for question answering. In EMNLP. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khot-2017-answering-complex-questions-open-information/