Elamur Research⌕
☰
home / papers / 2018 / The Web as a Knowledge-Base for Answering Complex Questions

The Web as a Knowledge-Base for Answering Complex Questions

Summary

The paper addresses complex open-domain questions that require combining several facts. The authors rely neither on a pre-built knowledge base nor on pre-supplied documents. A complex question is decomposed into a sequence of simpler ones, each of which is sent to a search engine. An RC model then finds the answer in web snippets, and the final answer is obtained through set operations such as union or intersection. To evaluate the approach, the ComplexWebQuestions dataset was created. It is based on SPARQL queries from WebQuestionsSP that were automatically made more complex by adding composition, conjunction, superlatives, and comparatives. Machine-generated questions were produced from these queries using templates, and Amazon Mechanical Turk workers rephrased them in natural language. The SplitQA model has a sequence-to-sequence architecture and maps a question to a short program, i.e., a computation tree. Its training data were obtained through noisy alignment of the machine-generated and human formulations. Decomposition raises precision@1 from 20.8 to 27.5, while a time-limited human reaches 63.0. The work shows that the web can replace a knowledge base for compositional questions. Decomposition helps primarily with retrieving relevant documents.

Key points

  • ComplexWebQuestions contains 34,689 question–answer pairs together with SPARQL queries and web snippets. For 462 KB predicates, 687 templates were manually annotated. The dataset is available at nlp.cs.tau.ac.il/compwebq, and the code is in the WebAsKB repository on GitHub.
  • The formal language consists of the functions SimpQA, Comp (substitutes answers for VAR), Conj (set intersection), and Add. Superlatives, comparatives, and negation are not decomposed but are passed to SimpQA directly. In the paper's running example, the answer is Lviv.
  • Results on ComplexWebQuestions (p@1):
    • SimpQA without decomposition — 20.8, SplitQA — 27.5, oracle — 33.7.
    • Another RC model also benefits from decomposition: SplitRCQA outperforms RCQA by 3.4 points.
    • The Google box gives the correct answer in only 2.5% of cases.
    • A human with a limit of 4 minutes per question reaches 63.0.
  • ComplexQuestions (Bao et al., 2016) is a small dataset for semantic parsing over Freebase:
    • SimpQA obtains 38.6 F1, and a simple rule that splits on the words "when/during" (SplitQARule) obtains 39.7 F1.
    • The best system, CompQ, which works directly with Freebase, achieves 40.9 F1.
    • On dev, SplitQARule yields 43.1 F1. After manually fixing string matches (SplitQARule++), the result rises to 46.9.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics: Question answering · Knowledge graphs & ontologies

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1059 · arXiv 1803.06643

@inproceedings{talmor2018web,
  title = {{The Web as a Knowledge-Base for Answering Complex Questions}},
  author = {Talmor, Alon and Berant, Jonathan},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1059},
  eprint = {1803.06643},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n18-1059}
}

References

44 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K. M. Hermann, T. Koisk, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS). find our card
  2. Y. Artzi and L. Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics (TACL) 1:49–62. find our card
  3. D. Hewlett, A. Lacoste, L. Jones, I. Polosukhin, A. Fandrianto, J. Han, M. Kelcey, and D. Berthelot. 2016. Wikireading: A novel large-scale language understanding task over Wikipedia. In Association for Computational Linguistics (ACL). find
  4. J. Bao, N. Duan, Z. Yan, M. Zhou, and T. Zhao. 2016. Constraint-based question answering with knowledge graph. In International Conference on Computational Linguistics (COLING). find
  5. J. Berant, A. Chou, R. Frostig, and P. Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  6. F. Hill, A. Bordes, S. Chopra, and J. Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. In International Conference on Learning Representations (ICLR). find our card
  7. J. Berant and P. Liang. 2015. Imitation learning of agenda-based semantic parsers. Transactions of the Association for Computational Linguistics (TACL) 3:545–558. find
  8. M. Iyyer, W. Yih, and M. Chang. 2016. Answering complicated question intents expressed in decomposed question sequences. CoRR 0. find
  9. K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In International Conference on Management of Data (SIGMOD). pages 1247–1250. find
  10. R. Jia and P. Liang. 2016. Data recombination for neural semantic parsing. In Association for Computational Linguistics (ACL). find
  11. R. Jia and P. Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  12. C. Buck, J. Bulian, M. Ciaramita, A. Gesmundo, N. Houlsby, W. Gajewski, and W. Wang. 2017. Ask the right questions: Active question reformulation with reinforcement learning. arXiv preprint arXiv:1705.07830 . — [arXiv:1705.07830](https://arxiv.org/abs/1705.07830) find
  13. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL). find our card
  14. D. Chen, J. Bolton, and C. D. Manning. 2016. A thorough examination of the CNN / Daily Mail reading comprehension task. In Association for Computational Linguistics (ACL). find our card
  15. D. Chen, A. Fisch, J. Weston, and A. Bordes. 2017. Reading Wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051 . find our card
  16. J. Krishnamurthy and T. Mitchell. 2012. Weakly supervised training of semantic parsers. In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL). pages 754–765. find
  17. K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 . find our card
  18. T. Kwiatkowski, E. Choi, Y. Artzi, and L. Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  19. C. Clark and M. Gardner. 2017. Simple and effective multi-paragraph reading comprehension. arXiv preprint arXiv:1710.10723 . find our card
  20. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683 . find our card
  21. J. Duchi, E. Hazan, and Y. Singer. 2010. Adaptive subgradient methods for online learning and stochastic optimization. In Conference on Learning Theory (COLT). find
  22. K. lee, M. Lewis, and L. Zettlemoyer. 2016. Global neural CCG parsing with optimality guarantees. In Empirical Methods in Natural Language Processing (EMNLP). find
  23. M. Dunn, , L. Sagun, M. Higgins, U. Guney, V. Cirik, and K. Cho. 2017. SearchQA: A new Q&A dataset augmented with context from a search engine. arXiv . find our card
  24. P. Liang. 2013. Lambda dependency-based compositional semantics. arXiv preprint arXiv:1309.4408 . find
  25. P. Liang, M. I. Jordan, and D. Klein. 2011. Learning dependency-based compositional semantics. In Association for Computational Linguistics (ACL). pages 590–599. find
  26. K. Guu, P. Pasupat, E. Z. Liu, and P. Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In Association for Computational Linguistics (ACL). find our card
  27. W. Yih, M. Richardson, C. Meek, M. Chang, and J. Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Association for Computational Linguistics (ACL). find our card
  28. T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Workshop on Cognitive Computing at NIPS. find our card
  29. M. Zelle and R. J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Association for the Advancement of Artificial Intelligence (AAAI). pages 1050–1055. find our card
  30. R. Nogueira and K. Cho. 2016. End-to-end goal-driven web navigation. In Advances in Neural Information Processing Systems (NIPS). find
  31. T. Onishi, H. Wang, M. Bansal, K. Gimpel, and D. McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Empirical Methods in Natural Language Processing (EMNLP). find
  32. L. S. Zettlemoyer and M. Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Uncertainty in Artificial Intelligence (UAI). pages 658– 666. find our card
  33. P. Pasupat and P. Liang. 2015. Compositional semantic parsing on semi-structured tables. In Association for Computational Linguistics (ACL). find our card
  34. V. Zhong, C. Xiong, and R. Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103 . find
  35. J. Pennington, R. Socher, and C. D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  36. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). find our card
  37. The Syntactic Process. MIT I. Sutskever, O. Vinyals, and Q. V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS). pages 3104–3112. find
  38. A. Talmor, M. Geva, and J. Berant. 2017. Evaluating semantic parsing against a simple web-based question answering model. In *SEM. find
  39. O. Vinyals, M. Fortunato, and N. Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems (NIPS). pages 2674–2682. find
  40. W. Wang, N. Yang, F. Wei, B. Chang, and M. Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Association for Computational Linguistics (ACL). find our card
  41. Y. Wang, J. Berant, and P. Liang. 2015. Building a semantic parser overnight. In Association for Computational Linguistics (ACL). find
  42. Y. Watanabe, B. Dhingra, and R. Salakhutdinov. 2017. Question answering from unstructured text by retrieval and comprehension. arXiv preprint arXiv:1703.08885 . — [arXiv:1703.08885](https://arxiv.org/abs/1703.08885) find
  43. J. Welbl, P. Stenetorp, and S. Riedel. 2017. Constructing datasets for multi-hop reading comprehension across documents. arXiv preprint arXiv:1710.06481 . find our card
  44. Y. Yang, W. Yih, and C. Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Empirical Methods in Natural Language Processing (EMNLP). pages 2013–2018. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/talmor-2018-web-as-knowledge-base-answering/