Elamur Research⌕
☰
home / papers / 2016 / Learning a Natural Language Interface with Neural Programmer

Learning a Natural Language Interface with Neural Programmer

Paper · 2016 · Arvind Neelakantan, Quoc V. Le, Martin Abadi et al.

Summary

The paper addresses answering natural-language questions over database tables without hand-built grammars, domain rules or annotated programs. The authors adapt Neural Programmer, a neural network with a built-in set of discrete table operations, to real-world data. At each step it chooses an operation and a column. Its components are an LSTM question encoder, a history RNN and a selector module. During training it uses soft, differentiable selection, so the latent program is learned by backpropagation from final answers only. At test time it uses hard selection.

The main change is to the objective function. Real data does not mark where an answer sits in the table, so the model must learn whether to compute an answer or look it up, and which matching cell is the right one. Strong dropout and weight decay limit overfitting.

On WikiTableQuestions, which has about 10,000 training examples and test tables never seen in training, a 15-model ensemble slightly beats a traditional semantic parser. The authors describe this as the first weakly supervised, end-to-end neural program inducer on a real-world dataset.

Key points

  • Data: WikiTableQuestions (Pasupat & Liang, 2015), with roughly 10,000 question–answer pairs as weak supervision. Test-time tables and column names are unseen during training.
  • Results: A single Neural Programmer model reaches 34.2% accuracy. A simply combined ensemble of 15 models reaches 37.7%, compared with 37.1% for the prior state-of-the-art semantic parser that relies on pruning.
  • Baselines: Sequence-to-sequence and pointer network baselines reach only single-digit accuracy.
  • Operations: The model uses 15 hand-defined operations chosen to mirror the baseline parser:
    • count, select, most frequent entry
    • argmax/argmin and comparisons
    • first/last/previous/next, print, reset
  • Program state: The program passes information between steps through a row selector variable. Answers are stored in scalar answer and lookup answer variables.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States533 works · 443,884 citations🎓University of Massachusetts AmherstAmherst Center · United States122 works · 69,574 citations🏢BaiduBeijing · China87 works · 17,840 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as Arvind Neelakantan, Quoc V. Le, Martin Abadi, Andrew McCallum and Dario Amodei. 2016. Learning a Natural Language Interface with Neural Programmer. arXiv:1611.08945.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1611.08945 · arXiv 1611.08945

@misc{neelakantan2016learning,
  title = {{Learning a Natural Language Interface with Neural Programmer}},
  author = {Arvind Neelakantan, Quoc V. Le, Martin Abadi, Andrew McCallum and Dario Amodei},
  year = {2016},
  doi = {10.48550/arxiv.1611.08945},
  eprint = {1611.08945},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1611.08945}
}

References

39 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh… — TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
  2. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. NAACL, 2016.
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2014.
  4. Antoine Bordes, Sumit Chopra, and Jason Weston. Question answering with subgraph embeddings. EMNLP, 2014. find
  5. Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. NIPS, 2015. find
  6. Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. NIPS, 2016.
  7. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. NIPS, 2015.
  8. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997. find
  9. Mohit Iyyer, Jordan L. Boyd-Graber, Leonardo Max Batista Claudino, Richard Socher, and Hal Daumé III. A neural network for factoid question answering over paragraphs. EMNLP, 2014. find
  10. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. Deep unordered composition rivals syntactic methods for text classification. ACL, 2015.
  11. Robin Jia and Percy Liang. Data recombination for neural semantic parsing. ACL, 2016. find
  12. Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. NIPS, 2015. find
  13. Lukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. ICLR, 2016. find
  14. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2014.
  15. Tomas Kocisky, Gabor Melis, Edward Grefenstette, Chris Dyer, Wang Ling, Phil Blunsom, and Karl Moritz Hermann. Semantic parsing with semi-supervised sequential autoencoders. ArXiv, 2016. find
  16. Jayant Krishnamurthy and Thomas Kollar. Jointly learning to parse and perceive: Connecting natural language to the physical world. TACL, 2013.
  17. Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. ICML, 2016.
  18. Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke Zettlemoyer. Scaling semantic parsers with on-the-fly ontology matching. EMNLP, 2013.
  19. Chen Liang, Jonathan Berant, Quoc Le, Kenneth Forbus, and Ni Lao. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. NAMPI Workshop, NIPS, 2016. find
  20. Percy Liang. Learning executable semantic parsers for natural language understanding. ACM, 2016. find
  21. Percy Liang, Michael I. Jordan, and Dan Klein. Learning dependency-based compositional semantics. ACL, 2011. find
  22. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. ArXiv, 2013.
  23. Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. Neural programmer: Neural programmer: Inducing latent programs with gradient descent. ICLR, 2016. find
  24. Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. ACL, 2015.
  25. Baolin Peng, Zhengdong Lu, Hang Li, and Kam-Fai Wong. Towards neural network-based reasoning. ArXiv, 2015. find
  26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. ArXiv, 2016. — SQuAD: 100,000+ Questions for Machine Comprehension of Text
  27. Scott Reed and Nando De Freitas. Neural programmer-interpreters. ICLR, 2016. find
  28. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 2014.
  29. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. NIPS, 2015. find
  30. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. NIPS, 2014.
  31. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. NIPS, 2015. find
  32. Shuohang Wang and Jing Jiang. Machine comprehension using match-lstm and answer pointer. ArXiv, 2016.
  33. Yushi Wang, Jonathan Berant, and Percy Liang. Building a semantic parser overnight. ACL, 2015. find
  34. P. Werbos. Backpropagation through time: what does it do and how to do it. Proceedings of IEEE, 1990. find
  35. Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 1992. find
  36. Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao. Neural enquirer: Learning to query tables with natural language. ArXiv, 2015. find
  37. Wojciech Zaremba, Tomas Mikolov, Armand Joulin, and Rob Fergus. Learning simple algorithms from examples. ICML, 2016. find
  38. John M. Zelle and Raymond J. Mooney. Learning to parse database queries using inductive logic programming. AAAI/IAAI, 1996.
  39. Luke S. Zettlemoyer and Michael Collins. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. UAI, 2005.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/neelakantan-2016-learning-natural-language-interface-neural/