Elamur Research⌕
☰
home / papers / 2019 / Question Answering by Reasoning Across Documents with Graph Convolutional Networks

Question Answering by Reasoning Across Documents with Graph Convolutional Networks

Paper · 2019 · Nicola De Cao, Wilker Aziz and Ivan Titov

Summary

Most reading comprehension systems answer questions from a single document or paragraph. This paper targets multi-hop question answering, where the answer can only be reached by chaining evidence spread over several documents. The authors propose Entity-GCN and evaluate it on the WikiHop dataset (built from WIKIPEDIA and WIKIDATA). For each query, they build a graph whose nodes are mentions of the candidate answers and the query subject. Mentions are found by exact string matching and by an external coreference system (Lee et al., 2017). The edges are typed: same-document co-occurrence, exact match, coreference, and a complement relation that keeps the graph connected. Node representations come from pre-computed ELMo embeddings. A relational GCN then passes messages over several hops before a feed-forward layer scores the candidates. Because no recurrent document encoder is trained, most of the computation happens offline and training is cheap. Entity-GCN beat the best previously published WikiHop results by more than 2%. It shows that graph-based propagation can replace costly query-aware document encoders, which matters for scaling QA to large document collections.

Key points

  • Model components: only a small query encoder, the R-GCN layers (Schlichtkrull et al., 2018) and a feed-forward answer selector are trained. ELMo supplies the initial node features.
  • Speed and robustness: training is at least 5× faster than the small BiDAF baseline. Even with plain pre-trained word embeddings instead of ELMo, the model stays competitive with RNN-based readers.
  • Results: the single model gains over 2% on prior published results. An ensemble adds another 3.6% and comes within about 3% of the human performance reported for WikiHop.
  • Validation analysis: the ensemble reaches 68.5% accuracy and the single model 65.3%. Easy relations include "member of political party" (85.5%). Hard ones include "place of birth" (51.0%) and "inception" (29.9%).
  • Error analysis: the main failure sources are mismatches between WIKIPEDIA and WIKIDATA and answers given at a different granularity (e.g., "London" vs "UK"). Accuracy falls as the number of candidates grows (Pearson −0.687), but it degrades more gently as the graph gains nodes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations🎓University of AmsterdamAmsterdam · Netherlands89 works · 81,337 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2 Reasoning · 510 works Question answering · 242 works

Cite

Cite as Nicola De Cao, Wilker Aziz and Ivan Titov. 2019. Question Answering by Reasoning Across Documents with Graph Convolutional Networks. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1240

@inproceedings{cao2019question,
  title = {{Question Answering by Reasoning Across Documents with Graph Convolutional Networks}},
  author = {Nicola {De Cao}, Wilker Aziz and Ivan Titov},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1240},
  url = {https://doi.org/10.18653/v1/n19-1240}
}

References

31 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Simaan. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1957–1967. Association for Computational Li… our card
  2. Lisa Bauer, Yicheng Wang, and Mohit Bansal. 2018. Commonsense for generative multi-hop question answering tasks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4220–4230. Association for Computational Linguistics. our card
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805. our card
  4. Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2018. Neural models for reasoning over multiple mentions using coreference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie… our card
  5. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701. our card
  6. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1601–1611. our card
  7. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. Kinga, D., and J. Ba Adam. ”A method for stochastic optimization.” International Conference on Learning Representations (ICLR)., 5. find
  8. Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations (ICLR). our card
  9. Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317– 328. our card
  10. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark. Association for Computational Ling… our card
  11. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197. Association for Computational Linguistics. our card
  12. Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1506–1515. Association for Computational Linguistics. our card
  13. Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2017. Reasonet: Learning to stop reading in machine comprehension. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1047–1055. ACM. find
  14. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. our card
  15. Linfeng Song, Zhiguo Wang, Mo Yu, Yue Zhang, Radu Florian, and Daniel Gildea. 2018. Exploring Graph-structured Passage Representation for Multi-hop Reading Comprehension with Graph Neural Networks. arXiv preprint arXiv:1809.02040. — [arXiv:1809.02040](https://arxiv.org/abs/1809.02040) find
  16. Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017. Cross-sentence n-ary relation extraction with graph lstms. Transactions of the Association for Computational Linguistics, 5:101–115. our card
  17. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. our card
  18. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… our card
  19. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. find
  20. Martin Raison, Pierre-Emmanuel Mazaré, Rajarshi Das, and Antoine Bordes. 2018. Weaver: Deep co-encoding of questions and documents for machine reading. In Proceedings of the International Conference on Machine Learning (ICML). find
  21. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. our card
  22. Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In The Semantic Web, pages 593– 607, Cham. Springer International Publishing. find
  23. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. our card
  25. Denny Vrandečić. 2012. Wikidata: A new platform for collaborative data collection. In Proceedings of the 21st International Conference on World Wide Web, pages 1063–1064. ACM. find
  26. Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Making neural qa as simple as possible but not simpler. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 271–280. Association for Computational Linguistics. find
  27. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287–302. our card
  28. Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. arXiv preprint arXiv:1611.01604. our card
  29. Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018a. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2205–2215. Association for Computational Linguistics. our card
  30. Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J Smola, and Le Song. 2018b. Variational reasoning for question answering with knowledge graph. The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18). find
  31. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. International Conference on Learning Representations (ICLR). our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cao-2019-question-answering-reasoning-across-documents/