Elamur Research⌕
☰
home / papers / 2016 / Harnessing Deep Neural Networks with Logic Rules

Harnessing Deep Neural Networks with Logic Rules

Summary

Deep neural networks learn mostly from labeled examples. This makes them data-hungry, hard to interpret, and hard to steer with human knowledge. Hu et al. propose a general way to inject declarative first-order logic rules into arbitrary architectures such as CNNs and RNNs. Rules are written in soft logic, so truth values are continuous, and each rule has a confidence weight. At every training step, a "teacher" distribution is built in closed form by projecting the network's predictions onto a rule-constrained space, following the posterior regularization principle. The "student" network is then trained to match both the gold labels and the teacher's soft outputs. Teacher and student evolve together, and the rule knowledge ends up in the student's weights, so the student can predict without evaluating rules. Unlabeled data can also feed the imitation term, which gives semi-supervised learning. The authors test a CNN on sentiment analysis and a BLSTM on CoNLL-2003 NER. With only one or two intuitive rules, they reach results comparable to the best systems of the time, without adding model parameters.

Key points

  • The method combines knowledge distillation with posterior regularization (Ganchev et al., 2010). Unlike classic distillation, the teacher is not pre-trained; both models are updated iteratively.
  • On CoNLL-2003 NER, the base BLSTM scores 89.55 F1. Adding bigram tag-transition rules lifts the teacher q to 91.11 F1. Adding a rule for entities in lists brings it to 91.18 F1 (student p: 89.93). This nearly matches Joint-NER-EL (91.20) and BLSTM-CRF of Ma and Hovy (91.21).
  • In low-resource sentiment experiments, the rule-enhanced teacher beats the base CNN by 2.6% with 5% of the training data. The gains are 2.3% at 10% and 2.0% at 30%. Adding unlabeled data improves accuracy further.
  • The iterative distillation clearly outperforms plain semi-supervised posterior regularization, which applies the rules only through unlabeled data. It also beats projecting a network onto the rules after training.
  • The NER setup uses the BIOES tagging scheme, GloVe initialization and Adadelta. The gap between teacher and student is larger for NER than for sentiment, because the NER rules link adjacent positions and multiple instances, which requires joint inference.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1228

@inproceedings{hu2016harnessing,
  title = {{Harnessing Deep Neural Networks with Logic Rules}},
  author = {Hu, Zhiting and Ma, Xuezhe and Liu, Zhengzhong and Hovy, Eduard and Xing, Eric},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1228},
  url = {https://doi.org/10.18653/v1/p16-1228}
}

References

45 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stephen H Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. 2015. Hinge-loss Markov random fields and probabilistic soft logic. arXiv preprint arXiv:1505.04406. — [arXiv:1505.04406](https://arxiv.org/abs/1505.04406) find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. Proc. of ICLR. find our card
  3. Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. 2015. Deep convolutional inverse graphics network. In Proc. of NIPS, pages 2530–2538. find
  4. Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332– 1338. find
  5. Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Proc. of KDD, pages 535–541. ACM. find
  6. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proc. of NAACL. find
  7. Jason PC Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional LSTM-CNNs. arXiv preprint arXiv:1511.08308. find our card
  8. Quoc V Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. Proc. of ICML. find our card
  9. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. JMLR, 12:2493– 2537. find our card
  10. James Foulds, Shachi Kumar, and Lise Getoor. 2015. Latent topic networks: A versatile probabilistic programming framework for topic models. In Proc. of ICML, pages 777–786. find
  11. Manoel VM França, Gerson Zaverucha, and Artur S dAvila Garcez. 2014. Fast relational learning using bottom clause propositionalization with artificial neural networks. Machine learning, 94(1):81–104. find
  12. Kuzman Ganchev, Joao Graça, Jennifer Gillenwater, and Ben Taskar. 2010. Posterior regularization for structured latent variable models. JMLR, 11:2001–2049. find
  13. Artur S d’Avila Garcez, Krysia Broda, and Dov M Gabbay. 2012. Neural-symbolic learning systems: foundations and applications. Springer Science & Business Media. find
  14. Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdelrahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. 2012. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine… find
  15. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. find our card
  16. Percy Liang, Hal Daumé III, and Dan Klein. 2008. Structure compilation: trading structure for features. In Proc. of ICML, pages 592–599. ACM. find
  17. Percy Liang, Michael I Jordan, and Dan Klein. 2009. Learning from measurements in exponential families. In Proc. of ICML, pages 641–648. ACM. find
  18. David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. 2016. Unifying distillation and privileged information. Prof. of ICLR. find
  19. Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. 2015. Joint named entity recognition and disambiguation. In Proc. of EMNLP. find
  20. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proc. of ACL. find our card
  21. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proc. of NIPS, pages 3111–3119. find
  22. Marvin Minksy. 1980. Learning meaning. Technical Report AI Lab Memo. Project MAC. MIT. find
  23. Anh Nguyen, Jason Yosinski, and Jeff Clune. 2015. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proc. of CVPR, pages 427–436. IEEE. find
  24. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proc. of KDD, pages 168–177. ACM. find
  25. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proc. of ACL, pages 115–124. find our card
  26. Matthew J Johnson, David Duvenaud, Alexander B Wiltschko, Sandeep R Datta, and Ryan P Adams. 2016. Structured VAEs: Composing probabilistic graphical models and variational autoencoders. arXiv preprint arXiv:1603.06277. — [arXiv:1603.06277](https://arxiv.org/abs/1603.06277) find
  27. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proc. of EMNLP, volume 14, pages 1532–1543. find
  28. Theofanis Karaletsos, Serge Belongie, Cornell Tech, and Gunnar Rätsch. 2016. Bayesian representation learning with oracle constraints. In Proc. of ICLR. find
  29. Yoon Kim. 2014. Convolutional neural networks for sentence classification. Proc. of EMNLP. find our card
  30. Diederik P Kingma and Max Welling. 2014. Auto-encoding variational Bayes. In Proc. of ICLR. find our card
  31. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proc. of NIPS, pages 1097–1105. find
  32. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. Proc. of ICML. find our card
  33. Danilo Jimenez Rezende, Shakir Mohamed, Ivo Danihelka, Karol Gregor, and Daan Wierstra. 2016. One-shot generalization in deep generative models. arXiv preprint arXiv:1603.05106. — [arXiv:1603.05106](https://arxiv.org/abs/1603.05106) find
  34. Matthew Richardson and Pedro Domingos. 2006. Markov logic networks. Machine learning, 62(1-2):107–136. find
  35. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489. find
  36. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. of EMNLP, volume 1631, page 1642. Citeseer. find
  37. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. Proc. of ICLR. find our card
  38. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proc. of CoNLL, pages 142– 147. Association for Computational Linguistics. find
  39. Geoffrey G Towell, Jude W Shavlik, and Michiel O No-ordewier. 1990. Refinement of approximate domain theories by knowledge-based neural networks. In Proceedings of the eighth National conference on Artificial intelligence, pages 861–866. Boston, MA. find
  40. Sida Wang and Christopher Manning. 2013. Fast dropout training. In Proc. of ICML, pages 118–126. find
  41. Bishan Yang and Claire Cardie. 2014. Context-aware learning for sentence-level sentiment analysis with posterior regularization. In Proc. of ACL, pages 325–335. find
  42. Wenpeng Yin and Hinrich Schutze. 2015. Multichannel variable-size convolution for sentence classification. Proc. of CONLL. find
  43. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. find our card
  44. Ye Zhang, Stephen Roller, and Byron Wallace. 2016. MGNC-CNN: A simple approach to exploiting multiple word embeddings for sentence classification. Proc. of NAACL. find
  45. Jun Zhu, Ning Chen, and Eric P Xing. 2014. Bayesian inference with posterior regularization and applications to infinite latent SVMs. JMLR, 15(1):1799–1847. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hu-2016-harnessing-deep-neural-networks-logic/