Elamur Research⌕
☰
home / papers / 2015 / Language to Code: Learning Semantic Parsers for If-This-Then-That Recipes

Language to Code: Learning Semantic Parsers for If-This-Then-That Recipes

Paper · 2015 · Chris Quirk, Raymond Mooney and Michel Galley

Summary

The paper addresses natural-language programming: automatically turning a user's English description of an if-this-then-that rule into an executable program. The authors collected a large corpus from ifttt.com. Each recipe pairs one trigger with one action, drawn from channels such as Gmail, Android or Philips hue. Each recipe's HTML form was converted into an abstract syntax tree (AST) under a hand-built grammar. The descriptions are short, often incomplete and sometimes non-English, so the training signal is very noisy.

The authors compared several parsers:

  • a retrieval baseline based on string edit distance;
  • a phrase-based SMT system following Andreas et al. (2013);
  • a synchronous approach;
  • KRISP-inspired methods that use log-linear classifiers instead of SVMs to predict grammar productions.

Loosely synchronous, classifier-based systems performed best. The strongest, posclass, sometimes matched the gold program more often than crowdsourced Mechanical Turk workers did. The work matters because it shows that noisy "found data" from the web can train language-to-code systems with little manual effort. It also introduces a new benchmark for semantic parsing.

Key points

  • The corpus has 114,408 recipe–description pairs, collected from ifttt.com in January 2015.
  • The recipes involve 781 functions across 160 channels: triggers account for 552 functions spread over 128 channels, and actions account for 229 functions spread over 99 channels.
  • Evaluation uses three measures: channel accuracy, channel+function accuracy, and F-measure over all AST productions. Results are reported on test subsets filtered by crowd judgments, such as removing non-English or unintelligible descriptions.
  • On the 758 recipes where at least 3 of 5 turkers agreed with the gold program, posclass reached about 81% channel and 71% function accuracy.
  • Recipe URLs and turker annotations were released at research.microsoft.com/lang2code.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓The University of Texas at AustinAustin · United States109 works · 56,942 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semantic parsing & meaning representation · 185 works

Cite

Cite as Chris Quirk, Raymond Mooney and Michel Galley. 2015. Language to Code: Learning Semantic Parsers for If-This-Then-That Recipes. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-1085

@inproceedings{quirk2015language,
  title = {{Language to Code: Learning Semantic Parsers for If-This-Then-That Recipes}},
  author = {Chris Quirk, Raymond Mooney and Michel Galley},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-1085},
  url = {https://doi.org/10.3115/v1/p15-1085}
}

References

24 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas, Andreas Vlachos, and Stephen Clark. 2013. Semantic parsing as machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 47–52, Sofia, Bulgaria, August. Association for Computational Linguistics. find
  2. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP-13).
  3. Taylor Berg-Kirkpatrick, Alexandre Bouchard-Côté, John DeNero, and Dan Klein. 2010. Painless unsupervised learning with features. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 582–590, Los Angeles, C…
  4. S.R.K. Branavan, Harr Chen, Luke S. Zettlemoyer, and Regina Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In Joint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Proce… find
  5. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of the 43nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 263–270, Ann Arbor, MI.
  6. J. Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37 – 46. find
  7. Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. 2006. Scalable inference and training of context-rich syntactic translation models. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the…
  8. Sumit Gulwani and Mark Marron. 2014. Nlyze: Interactive programming by natural language for spreadsheet data analysis and manipulation. In SIGMOD. find
  9. Rohit J. Kate and Raymond J. Mooney. 2006. Using string-kernels for learning semantic parsers. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 913–920, Sydney, Australia, July. Associatio…
  10. R. J. Kate, Y. W. Wong, and R. J. Mooney. 2005. Learning to transform natural to formal languages. In Proceedings of the Twentieth National Conference on Artificial Intelligence (AAAI-05), pages 1062–1068, Pittsburgh, PA, July. find
  11. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03, pages 48–54, Stroudsburg, PA, USA. As… find
  12. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Annu… find
  13. Klaus Krippendorff. 1980. Content Analysis: an Introduction to its Methodology. Sage Publications, Beverly Hills, CA. find
  14. Vu Le, Sumit Gulwani, and Zhendong Su. 2013. Smartsynth: Synthesizing smartphone automation scripts from natural language. In MobiSys. find
  15. Greg Little and Robert C. Miller. 2007. Keyword programming in java. In Proceedings of the Twenty-second IEEE/ACM International Conference on Automated Software Engineering, ASE ’07, pages 84– 93, New York, NY, USA. ACM. find
  16. Huma Lodhi, Craig Saunders, John Shawe-Taylor, Nello Cristianini, and Chris Watkins. 2002. Text classification using string kernels. Journal of Machine Learning Research, 2:419–444. find
  17. Franz Josef Och, Christoph Tillmann, and Hermann Ney. 1999. Improved alignment models for statistical machine translation. In Proc. of the Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 20–28, University of Maryland, College Park, MD, June.
  18. Yuk Wah Wong and Raymond Mooney. 2006. Learning for semantic parsing with statistical machine translation. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 439–446, New York City, USA, June. Association for Computational Linguistics.
  19. Yuk Wah Wong and Raymond J. Mooney. 2007a. Generation by inverting a semantic parser that uses statistical machine translation. In Proceedings of Human Language Technologies: The Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 172– 179, Ro…
  20. Yuk Wah Wong and Raymond J. Mooney. 2007b. Learning synchronous grammars for semantic parsing with lambda calculus. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL), pages 960–967, Prague, Czech Republic, June.
  21. William A. Woods. 1977. Lunar rocks in natural English: Explorations in natural language question answering. In Antonio Zampoli, editor, Linguistic Structures Processing. Elsevier North-Holland, New York. find
  22. Dekai Wu. 1997. Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Computational Linguistics, 23(3):377–403.
  23. John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI-96), pages 1050–1055, Portland, OR, August.
  24. Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In In Proceedings of the 21st Conference on Uncertainty in AI, pages 658–666.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/quirk-2015-language-code-learning-semantic-parsers/