Elamur Research⌕
☰
home / papers / 2018 / Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge

Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge

Summary

Cloze-style reading comprehension asks a system to restore a blanked-out word in a question sentence using an accompanying story. Often the deciding clue is everyday knowledge the text never states, for example that people mount horses rather than swords. The authors extend the single-pass Attention Sum Reader (AS Reader) into the Knowledgeable Reader (Kn-Reader).

A heuristic retrieval step scores ConceptNet triples by whether their lemmas match answer candidates, question tokens or document tokens. It keeps the same number of facts for each candidate. The selected facts go into a key-value memory. Every story and question token attends over this memory, and the resulting knowledge vector is merged with its BiGRU context encoding before answer scoring. Several text-only and knowledge-enriched interactions are then combined in an ensemble.

On the hard Common Nouns data, the added knowledge lowers relative error by about 9%. This lets a simple one-hop model compete with much more elaborate multi-hop architectures, and it also gives a slight gain on named entities. Because attention over facts can be inspected, the model reveals which background knowledge supported each prediction.

Key points

  • Knowledge source: the Open Mind Common Sense (OMCS) part of ConceptNet, roughly 630k (subject, relation, object) triples. Three variants were tested: CN5All, CN5Sel (some WordNet relations removed) and CN5WN3 (WordNet-sourced facts only).
  • Retrieval weights matched lemmas by where they occur: 4 for answer candidates, 3 for the question and 2 for the document. Memory sizes of 50, 100 or 200 facts per instance were tried, with facts spread evenly across the 10 candidates to avoid frequency bias.
  • Architecture details:
    • The question is represented by the BiGRU state at the placeholder position rather than by final states.
    • The hidden size is 256 per GRU direction.
    • Embedding dropout uses a keep probability of 0.8.
  • Ablation and attention analysis show which interactions matter most. Enriching only the question (D ctx, Q ctx+kn) contributes most, and enriching both question and document helps when the story context is needed.
  • Case studies show the model can override frequency bias toward wrong candidates and link non-trivial paraphrases (e.g., undressed–naked). Gaps remain in event-related knowledge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Question answering · Reasoning

Institutions

🎓Heidelberg UniversityHeidelberg · Germany20 works · 40 006 citations🎓Technische Universität DarmstadtDarmstadt · Germany58 works · 15 788 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1076

@inproceedings{mihaylov2018knowledgeable,
  title = {{Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge}},
  author = {Mihaylov, Todor and Frank, Anette},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1076},
  url = {https://doi.org/10.18653/v1/p18-1076}
}

References

42 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28,… find our card
  2. Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, and Yoshua Bengio. 2016. A neural knowledge language model. In CoRR, volume abs/1608.00318. find
  3. Nikita Bhutani, H V Jagadish, and Dragomir Radev. 2016. Nested propositions in open information extraction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 55–64, Austin, Texas. Association for Computational Linguistics. find
  4. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. volume abs/1511.02301. find our card
  5. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems 26, pages 2787–2795. Curran Associates, Inc. find
  6. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver… find our card
  7. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358–2367, Berlin, Germany. Association for C… find our card
  8. Rudolf Kadlec, Martin Schmid, Ondřej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 908–918. Association for Computational Linguist… find our card
  9. Junyoung Chung, Çalar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv e-prints, abs/1412.3555. Presented at the Deep Learning workshop at NIPS2014. find
  10. Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi. 2016. Globally coherent text generation with neural checklist models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 329–339, Austin, Texas. Association for Computational Linguistics. find our card
  11. Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017. Attention-over-attention neural networks for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 593–602. Association for Comput… find
  12. Teng Long, Emmanuel Bengio, Ryan Lowe, Jackie Chi Kit Cheung, and Doina Precup. 2017. World knowledge for reading comprehension: Rare entity prediction with hierarchical lstms using external descriptions. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, page… find
  13. Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers for text comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1832–1846. Association for Computational… find
  14. Mausam, Michael Schmitz, Stephen Soderland, Robert Bart, and Oren Etzioni. 2012. Open language learning for information extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 523–534, Jeju Island,… find
  15. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1535–1545, Edinburgh, Scotland, UK. Association for Computational Linguistics. find
  16. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1400–1409, Austin, Texas. Association for… find
  17. Shizhu He, Cao Liu, Kang Liu, and Jun Zhao. 2017. Generating natural answers by incorporating copying and retrieving mechanisms in sequence-to-sequence learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 199– 208. Associ… find
  18. George A. Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J. Miller. 1990. Introduction to wordnet: An on-line lexical database*. In International Journal of Lexicography, volume 3, pages 235–244. of the Workshop on Cognitive Computation: Integrating neural and symbolic app… find
  19. T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, J. Betteridge, A. Carlson, B. Dalvi, M. Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Platanios, A. Ritter, M. Samadi, B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen, A. Saparov, M. Greaves, and J. Welling.… find
  20. Parmjit Singh, T Lin, E.T. Mueller, G Lim, T Perkins, and W.L. Zhu. 2002. Open mind common sense: Knowledge acquisition from the general public. In Lecture Notes in Computer Science, volume 2519, pages 1223–1237. find
  21. Sameer Singh, Amarnag Subramanya, Fernando Pereira, and Andrew McCallum. 2012. Wikilinks: A large-scale cross-document coreference corpus labeled via links to Wikipedia. Technical Report UM-CS-2012-015. find
  22. Tsendsuren Munkhdalai and Hong Yu. 2016. Reasoning with Memory Augmented Neural Networks for Language Comprehension. In International Conference on Learning Representations (ICLR) 2017. find
  23. Alessandro Sordoni, Phillip Bachman, and Yoshua Bengio. 2016. Iterative alternating neural attention for machine reading. abs/1606.02245. — [arXiv:1606.02245](https://arxiv.org/abs/1606.02245) find
  24. Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217– 250. find
  25. Robert Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. In AAAI. find
  26. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2230– 2235, Austin, Texas. Association for Computational Lingu… find
  27. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. volume 15, pages 1929–1958. find
  28. Thomas Pellissier Tanon, Denny Vrande, San Francisco, Sebastian Schaffert, and Thomas Steiner. 2016. From Freebase to Wikidata : The Great Migration. In Proceedings of the 25th International Conference on World Wide Web, pages 1419–1428. find
  29. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. Association for Computational Linguistics. find our card
  30. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200, Vancouver, Canada. Association for Computational Linguis… find our card
  31. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. find our card
  32. Adam Trischler, Zheng Ye, Xingdi Yuan, Philip Bachman, Alessandro Sordoni, and Kaheer Suleman. 2016. Natural language comprehension with the epireader. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 128–137. Association for Computational Linguistics. find
  33. Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193–203, Seattle, Washington, USA. Association for Comp… find our card
  34. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hananneh Hajishirzi. 2017. Bi-Directional Attention Flow for Machine Comprehension. In Proceedings of International Conference of Learning Representations 2017, pages 1–12. find
  35. Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 189–198. Association for Comput… find our card
  36. Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2016. Reasonet: Learning to stop reading in machine comprehension. In Proceedings Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. 2016. Multi-perspective context matching for machine comprehension. CoRR, abs/1612.04211. — [arXiv:1612.04211](https://arxiv.org/abs/1612.04211) find
  37. Dirk Weissenborn, Tomas Kocisky, and Chris Dyer. 2017. Dynamic integration of background knowledge in neural NLU systems. CoRR, abs/1706.02596. — [arXiv:1706.02596](https://arxiv.org/abs/1706.02596) find
  38. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. In International Conference on Learning Representations (ICLR), 2015. find
  39. Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. In International Conference on Learning Representations (ICLR), 2017, volume abs/1611.01604. find our card
  40. Bishan Yang and Tom Mitchell. 2017. Leveraging knowledge bases in lstms for improving machine reading. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1436–1446. Association for Computational Linguistics. find
  41. Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In International Conference on Learning Representations (ICLR), 2015. find our card
  42. Zichao Yang, Phil Blunsom, Chris Dyer, and Wang Ling. 2017. Reference-aware language models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1851–1860. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mihaylov-2018-knowledgeable-reader-enhancing-cloze-style/