Elamur Research⌕
☰
home / papers / 2018 / Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback

Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback

Paper · 2018 · Carolin Lawrence and Stefan Riezler

Summary

Building training data for semantic parsers is costly because expert-written parses are hard to obtain. Gold answers are also hard to define for fuzzy geographical questions over OpenStreetMap (OSM), such as "near" or walking-distance queries. Lawrence and Riezler explore a weaker signal. A deployed parser proposes a single query per question, a user judges it, and these logged judgments later train a new parser offline. This setting is counterfactual learning from human bandit feedback. To collect feedback, they built an interface that turns each parse into simple statements that non-experts can mark as right or wrong, which also yields token-level rewards. On the learning side, they adapt the deterministic propensity matching (DPM) objective to a neural sequence-to-sequence parser. They add a one-step-late (OSL) reweighting that makes self-normalization compatible with mini-batch SGD. Combining OSL with token-level rewards (DPM+T+OSL) significantly beats both the supervised baseline and a bandit-to-supervised (B2S) approach, on human and larger simulated logs. This suggests commercial assistants could improve their parsers from cheap user interaction logs.

Key points

  • Human judges needed about 16.4 seconds on average to give feedback on one question–parse pair.
  • With human feedback, answer F1 rose from 57.45 (baseline) to 58.44 with DPM+T+OSL. B2S reached only 57.79.
  • With a larger simulated log, DPM+T+OSL reached 64.41 F1 (+6.96), significantly ahead of B2S at 63.22. In this log, only 46.27% of logged queries were fully correct, so token-level rewards let the model learn from partially correct queries.
  • Most queries fixed by the improved parser were corrected OSM tags: 90% with human feedback and 86.75% with simulated feedback. A smaller share involved question type or query structure.
  • Recomputing the OSL reweighting constant four times per epoch worked best. Updating it after every mini-batch was too slow to be practical.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Heidelberg UniversityHeidelberg · Germany38 works · 41,356 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksDecision Sciences · 209 worksManagement Science and Operations Research · 108 works

🧭 Topics 1Research subjects and languages the work is about.Semantic parsing & meaning representation · 185 works

Cite

Cite as Carolin Lawrence and Stefan Riezler. 2018. Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1169

@inproceedings{lawrence2018improving,
  title = {{Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback}},
  author = {Carolin Lawrence and Stefan Riezler},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1169},
  url = {https://doi.org/10.18653/v1/p18-1169}
}

References

35 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1(1).
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), San Diego, CA.
  3. Léon Bottou, Jonas Peters, Joaquin Qui nonero Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. 2013. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14. find
  4. Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langu…
  5. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. ArXiv e-prints, 1412.3555. find
  6. Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), Berlin, Germany.
  7. Miroslav Dudik, John Langford, and Lihong Li. 2011. Doubly robust policy evaluation and learning. In Proceedings of the 28th International Conference on Machine Learning (ICML), New York, NY. find
  8. Michael C. Fu. 2006. Gradient estimation. Handbook in Operations Research and Management Science, 13. find
  9. Dan Goldwasser and Dan Roth. 2013. Learning from natural instructions. Machine Learning, 94(2). find
  10. Peter J. Green. 1990. On the use of the EM algorithm for penalized likelihood estimation. Journal of the Royal Statistical Society B, 52(3). find
  11. Carolin Haas and Stefan Riezler. 2016. A corpus and semantic parser for multilingual natural language querying of openstreetmap. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), San Diego, Calif… find
  12. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. find
  13. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver, Canada. find
  14. Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), Berlin, Germany. find
  15. Nan Jiang and Lihong Li. 2016. Doubly robust off-policy value evaluation for reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning (ICML), New York, New York, USA. find
  16. Thorsten Joachims, Adith Swaminathan, and Maarten de Rijke. 2018. Deep learning with logged bandit feedback. In International Conference on Learning Representations (ICLR). find
  17. Augustine Kong. 1992. A note on importance sampling using standardized weights. Technical Report 348, Department of Statistics, University of Chicago, Illinois. find
  18. Carolin Lawrence, Pratik Gajane, and Stefan Riezler. 2017a. Counterfactual learning for machine translation: Degeneracies and solutions. In Proceedings of the NIPS WhatIF Workshop, Long Beach, CA. find
  19. Carolin Lawrence and Stefan Riezler. 2016. Nlmaps: A natural language interface to query openstreetmap. In Proceedings of the 26th International Conference on Computational Linguistics: System Demonstrations (COLING), Osaka, Japan. find
  20. Carolin Lawrence, Artem Sokolov, and Stefan Riezler. 2017b. Counterfactual learning from bandit feedback under deterministic logging : A case study in statistical machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), Copenhagen, Denma… find
  21. Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus, and Ni Lao. 2017. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver, Canada. find
  22. Lili Mou, Zhengdong Lu, Hang Li, and Zhi Jin. 2017. Coupling distributed and symbolic execution for natural language queries. In Proceedings of the 34th International Conference on Machine Learning (ICML), Sydney, Australia. find
  23. Arvind Neelakantan, Quoc V. Le, Martı́n Abadi, Andrew McCallum, and Dario Amodei. 2017. Learning a natural language interface with neural programmer. In International Conference on Learning Representations (ICLR), Toulon, France. find
  24. Eric W. Noreen. 1989. Computer Intensive Methods for Testing Hypotheses: An Introduction. Wiley, New York. find
  25. Haoruo Peng, Ming-Wei Chang, and Wen-tau Yih. 2017. Maximum margin reward networks for learning from explicit and implicit supervision. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), Copenhagen, Denmark. find
  26. Doina Precup, Richard S. Sutton, and Satinder P. Singh. 2000. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning (ICML), San Francisco, CA, USA. find
  27. Paul R. Rosenbaum and Donald B. Rubin. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1). find
  28. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine translation. In Proceedings of the Software Demonstrations of…
  29. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS), Montreal, Canada.
  30. Adith Swaminathan and Thorsten Joachims. 2015a. Batch learning from logged bandit feedback through counterfactual risk minimization. Journal of Machine Learning Research, 16. find
  31. Adith Swaminathan and Thorsten Joachims. 2015b. The self-normalized estimator for counterfactual learning. In Advances in Neural Information Processing Systems (NIPS), Montreal, Canada. find
  32. Philip Thomas and Emma Brunskill. 2016. Data-efficient off-policy policy evaluation for reinforcement learning. In Proceedings of the 33nd International Conference on Machine Learning (ICML), New York, NY. find
  33. Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 20. find
  34. Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL).
  35. Matthew D. Zeiler. 2012. ADADELTA: An adaptive learning rate method. ArXiv e-prints, 1212.5701.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lawrence-2018-improving-neural-semantic-parser-counterfactual/