Elamur Research⌕
☰
home / papers / 2016 / A Neural Knowledge Language Model

A Neural Knowledge Language Model

Paper · 2016 · Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa et al.

Summary

Conventional neural language models are poor at conveying factual knowledge. Names of people, places, and dates occur rarely in a corpus. As a result, the model either lacks such words in its vocabulary or substitutes the <unk> token for them. The authors propose the Neural Knowledge Language Model (NKLM), an LSTM-based RNN model combined with a knowledge graph. At each step, the model first determines whether the next word corresponds to some fact about the topic. It then either takes the word from the regular vocabulary or copies it from the name of that fact's object. This allows the model to produce words it did not see during training. To train it, the authors built the WikiFact dataset, in which Wikipedia descriptions are automatically aligned with facts from Freebase. They also proposed the Unknown-Penalized Perplexity metric, which prevents a model from getting a good score merely because it predicts <unk> accurately. NKLM outperforms the baseline RNNLM on all three perplexity variants and generates far fewer unknown words. The work is an early example of combining symbolic knowledge with a neural language model through a copy mechanism.

Key points

  • WikiFact: Wikipedia texts aligned with Freebase facts by simple string matching. Words with no fact behind them (e.g., function words) get a separate class, Not-a-Fact (NaF).
  • Fact representation: each fact is a (subject, relation, object) triple with 100-dimensional TransE embeddings. These vectors are kept fixed during training, so facts can be updated without retraining the model. When the place of birth Oklahoma was replaced with Chicago, the model immediately wrote the new city and left the rest of the text unchanged.
  • Setup: both models are two-layer LSTMs with dropout and 1000 hidden units. The baseline RNNLM has a 10K-word vocabulary. NKLM has slightly fewer parameters than RNNLM.
  • New metrics UPP and UPP-fact: the probability of <unk> is divided by the number of out-of-vocabulary words. In UPP these are all such words in the corpus; in UPP-fact, only the knowledge words about the current topic.
  • The main source of the gain is word copying: without it, NKLM is only slightly better than RNNLM. Limitation: the model does not use facts it has not recognized. For example, it writes "he" about a woman even though a gender fact is available.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa and Yoshua Bengio. 2016. A Neural Knowledge Language Model. arXiv:1608.00318.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1608.00318 · arXiv 1608.00318

@misc{ahn2016neural,
  title = {{A Neural Knowledge Language Model}},
  author = {Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa and Yoshua Bengio},
  year = {2016},
  doi = {10.48550/arxiv.1608.00318},
  eprint = {1608.00318},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1608.00318}
}

References

35 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
  2. Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian, Bergeron, Arnaud, Bouchard, Nicolas, Warde-Farley, David, and Bengio, Yoshua. Theano: new features and speed improvements. arXiv preprint arXiv:1211.5590, 2012.
  3. Bengio, Yoshua, Ducharme, Réjean, Vincent, Pascal, and Jauvin, Christian. A neural probabilistic language model. In Journal of Machine Learning Research, 2003.
  4. Blei, David M, Ng, Andrew Y, and Jordan, Michael I. Latent dirichlet allocation. the Journal of machine Learning research, 3:993–1022, 2003.
  5. Bordes, Antoine, Usunier, Nicolas, Garcia-Duran, Alberto, Weston, Jason, and Yakhnenko, Oksana. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems, pp. 2787–2795, 2013.
  6. Bordes, Antoine, Usunier, Nicolas, Chopra, Sumit, and Weston, Jason. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075, 2015. find
  7. Celikyilmaz, Asli, Hakkani-Tur, Dilek, Pasupat, Panupong, and Sarikaya, Ruhi. Enriching word embeddings using knowledge graph for semantic tagging in conversational dialog systems. In 2015 AAAI Spring Symposium Series, 2015. find
  8. Chelba, Ciprian, Mikolov, Tomas, Schuster, Mike, Ge, Qi, Brants, Thorsten, Koehn, Phillipp, and Robinson, Tony. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013.
  9. Gildea, Daniel and Hofmann, Thomas. Topic-based language models using em. EuroSpeech 1999, 1999. find
  10. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. find
  11. Gu, Jiatao, Lu, Zhengdong, Li, Hang, and Li, Victor O. K. Incorporating copying mechanism in sequence-to-sequence learning. CoRR, abs/1603.06393, 2016.
  12. Gulcehre, Caglar, Ahn, Sungjin, Nallapati, Ramesh, Zhou, Bowen, and Bengio, Yoshua. Pointing the unknown words. ACL 2016, 2016.
  13. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  14. Iyyer, Mohit, Boyd-Graber, Jordan L, Claudino, Leonardo Max Batista, Socher, Richard, and Daumé III, Hal. A neural network for factoid question answering over paragraphs. In EMNLP 2014, pp. 633–644, 2014. find
  15. Jean, Sebastien, Cho, Kyunghyun, Memisevic, Roland, and Bengio, Yoshua. On using very large target vocabulary for neural machine translation. ACL 2015, 2015.
  16. Jozefowicz, Rafal, Vinyals, Oriol, Schuster, Mike, Shazeer, Noam, and Wu, Yonghui. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. find
  17. Kim, Yoon. Convolutional neural networks for sentence classification. EMNLP 2014, 2014.
  18. Lebret, Rémi, Grangier, David, and Auli, Michael. Neural text generation from structured data with application to the biography domain. arXiv preprint arXiv:1603.07771, 2016.
  19. Long, Teng, Lowe, Ryan, Cheung, Jackie Chi Kit, and Precup, Doina. Leveraging lexical resources for learning entity embeddings in multi-relational data. 2016. find
  20. Marcus, Mitchell P, Marcinkiewicz, Mary Ann, and Santorini, Beatrice. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330, 1993.
  21. Merity, Stephen, Xiong, Caiming, Bradbury, James, and Socher, Richard. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
  22. Mikolov, Tomas and Zweig, Geoffrey. Context dependent recurrent neural network language model. In Spoken Language Technology Workshop (SLT), 2012 IEEE, pp. 234–239. find
  23. Mikolov, Tomas, Karafiát, Martin, Burget, Lukas, Cernockỳ, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In INTERSPEECH 2010, volume 2, pp. 3, 2010. find
  24. Miller, George A. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41, 1995.
  25. Mnih, Andriy and Hinton, Geoffrey E. A scalable hierarchical distributed language model. In Advances in neural information processing systems, pp. 1081–1088, 2009. find
  26. Mnih, Andriy and Teh, Yee Whye. A fast and simple algorithm for training neural probabilistic language models. ICML 2012, 2012.
  27. Morin, Frederic and Bengio, Yoshua. Hierarchical probabilistic neural network language model. AISTATS 2005, pp. 246, 2005. find
  28. Nickel, Maximilian, Murphy, Kevin, Tresp, Volker, and Gabrilovich, Evgeniy. A review of relational machine learning for knowledge graphs: From multi-relational link prediction to automated knowledge graph construction. arXiv preprint arXiv:1503.00759, 2015. find
  29. Reed, Scott and de Freitas, Nando. Neural programmer-interpreters. ICLR 2016, 2016. find
  30. Serban, Iulian V, Sordoni, Alessandro, Bengio, Yoshua, Courville, Aaron, and Pineau, Joelle. Building end-to-end dialogue systems using generative hierarchical neural networks. 30th AAAI Conference on Artificial Intelligence, 2015. find
  31. Vinyals, Oriol and Le, Quoc. A neural conversational model. arXiv preprint arXiv:1506.05869, 2015.
  32. Vinyals, Oriol, Fortunato, Meire, and Jaitly, Navdeep. Pointer networks. NIPS 2015, 2015. find
  33. Weston, Jason, Chopra, Sumit, and Bordes, Antoine. Memory networks. ICLR 2015, 2015. find
  34. Weston, Jason, Bordes, Antoine, Chopra, Sumit, and Mikolov, Tomas. Towards ai-complete question answering: A set of prerequisite toy tasks. ICLR 2016, 2016.
  35. Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ahn-2016-neural-knowledge-language-model/