Elamur Research⌕
☰
home / papers / 2022 / Neurosymbolic Programming for Science

Neurosymbolic Programming for Science

Paper · 2022 · Jennifer J. Sun, Megan Tjandrasuwita, Atharva Sehgal et al.

Summary

This position paper, presented at the NeurIPS 2022 AI for Science Workshop, asks how Neurosymbolic Programming (NP) could bring AI closer to being an automated scientist. Such a system would propose and test hypotheses while remaining consistent with established knowledge. The authors frame a hypothesis as a program written in a Domain Specific Language (DSL). Experts build known features and constraints into the DSL, and neurosymbolic program synthesis then searches over program structures and fits their continuous parameters, often by using differentiable programs.

The paper reports no new experiments or datasets. Instead, it aligns the NP lifecycle with the stages of the scientific process. Its running example is animal behavior analysis from video: pose tracking followed by behavior classification. This domain was chosen because its spatiotemporal, noisy and lab-specific data are typical of many sciences. The authors then list open problems: imperfect raw data, encoding domain knowledge, scalable and stable learning, measuring interpretability, cross-domain benchmarks and user-facing tools. The work serves as a research agenda that invites ML and domain scientists to build interpretable, knowledge-aware models together.

Key points

  • NP models are placed on a neural-to-symbolic spectrum, shown with four behavior-analysis examples:
    • a 1D Convolutional Network,
    • a Neurosymbolic Encoder (Zhan et al., 2021),
    • a Differentiable Program (Tjandrasuwita et al., 2021),
    • a hand-written rule that detects a fly attacking a target.
  • Effective NP needs a well-designed DSL and scalable learning. Learning splits into architecture search (comparable to neural architecture search) and parameter optimization within a fixed architecture.
  • The Morlet Filter is a differentiable temporal operation that replaces generic map/fold. It is data-efficient and can be read as an impulse response to a behavioral feature.
  • The authors propose benchmarks built on low-dimensional spatiotemporal data. They identify several obstacles:
    • evaluation depends on costly expert review,
    • programs found by stochastic search may not be reproducible,
    • no agreed metric exists for robust reproducibility,
    • neural training is GPU-heavy, while program synthesis is CPU-heavy.
  • MARS and Bento (Segalin et al., 2021) are cited as models for tooling, and the authors suggest wrapping NEAR in a visual interface. The work was funded by NSF Grant #1918839.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States274 works · 82,013 citations🎓The University of Texas at AustinAustin · United States126 works · 58,144 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States80 works · 32,402 citations🎓California Institute of TechnologyPasadena · United States16 works · 1,291 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

Cite

Cite as Jennifer J. Sun, Megan Tjandrasuwita, Atharva Sehgal, Armando Solar-Lezama, Swarat Chaudhuri, Yisong Yue and Omar Costilla-Reyes. 2022. Neurosymbolic Programming for Science. arXiv:2210.05050.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2210.05050 · arXiv 2210.05050

@misc{sun2022neurosymbolic,
  title = {{Neurosymbolic Programming for Science}},
  author = {Jennifer J. Sun, Megan Tjandrasuwita, Atharva Sehgal, Armando Solar-Lezama, Swarat Chaudhuri, Yisong Yue and Omar Costilla-Reyes},
  year = {2022},
  doi = {10.48550/arxiv.2210.05050},
  eprint = {2210.05050},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2210.05050}
}

References

40 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David J Anderson and Pietro Perona. Toward a science of computational ethology. Neuron, 84(1): 18–31, 2014. find
  2. Matthew Bowers, Theo X. Olausson, Catherine Wong, Gabriel Grand, Joshua B. Tenenbaum, Kevin Ellis, and Armando Solar-Lezama. Top-down synthesis for library learning. Proc. ACM Program. Lang., (POPL), 2023. find
  3. Swarat Chaudhuri and Armando Solar-Lezama. Smooth interpretation. ACM Sigplan Notices, 45(6): 279–291, 2010. find
  4. Swarat Chaudhuri, Kevin Ellis, Oleksandr Polozov, Rishabh Singh, Armando Solar-Lezama, Yisong Yue, et al. Neurosymbolic programming. Foundations and Trends® in Programming Languages, 7 (3):158–243, 2021. find
  5. Miles Cranmer. Pysr: Fast & parallelized symbolic regression in python/julia, 2020. find
  6. Miles Cranmer, Alvaro Sanchez Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering Symbolic Models from Deep Learning with Inductive Biases. In Advances in Neural Information Processing Systems, volume 33, pages 17429–17442. Curran Associates, Inc., 2020.
  7. Guofeng Cui and He Zhu. Differentiable synthesis of program architectures. Advances in Neural Information Processing Systems, 34:11123–11135, 2021. find
  8. Sandeep Robert Datta, David J Anderson, Kristin Branson, Pietro Perona, and Andrew Leifer. Computational neuroethology: a call to action. Neuron, 104(1):11–24, 2019. find
  9. Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. RobustFill: Neural Program Learning under Noisy I/O. In Proceedings of the 34th International Conference on Machine Learning, pages 990–998. PMLR, July 2017. ISSN: 2640-3498. find
  10. Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017.
  11. Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B. Tenenbaum. DreamCoder: bootstrapping inductive program synthesis with wake-sleep library learning. In Proceedings of the 42nd ACM SIGPLAN International Conference… find
  12. Kevin Ellis, Adam Albright, Armando Solar-Lezama, Joshua B Tenenbaum, and Timothy J O’Donnell. Synthesizing theories of human language with bayesian program induction. Nature communications, 13(1):1–13, 2022. find
  13. Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997–2017, 2019. find
  14. Eyrun Eyjolfsdottir, Steve Branson, Xavier P. Burgos-Artizzu, Eric D. Hoopfer, Jonathan Schor, David J. Anderson, and Pietro Perona. Detecting Social Actions of Fruit Flies. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, Lecture Notes in Comp… — [doi:10.1007/978-3-319-10605-2_50](https://doi.org/10.1007/978-3-319-10605-2_50) find
  15. Joseph Garner. Mouse Ethogram – Stanford School of Medicine. URL https://mousebehavior.org/. Noah D. Goodman, Tomer D. Ullman, and Joshua B. Tenenbaum. Learning a theory of causality. Psychological Review, 118(1):110–119, January 2011. ISSN 1939-1471. doi: 10.1037/a0021336. — [doi:10.1037/a0021336](https://doi.org/10.1037/a0021336) ↗
  16. Nastacia L Goodwin, Simon RO Nilsson, Jia Jie Choong, and Sam A Golden. Toward the explainability, transparency, and universality of machine learning for behavioral classification in neuroscience. Current Opinion in Neurobiology, 73:102544, 2022. find
  17. Sumit Gulwani. Automating String Processing in Spreadsheets using Input-Output Examples. In PoPL’11, January 26-28, 2011, Austin, Texas, USA, January 2011.
  18. Minghao Guo, Veronika Thost, Beichen Li, Payel Das, Jie Chen, and Wojciech Matusik. Data-efficient graph grammar learning for molecular generation. In International Conference on Learning Representations, 2021. find
  19. Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. Human decisions and machine predictions. The quarterly journal of economics, 133(1):237–293, 2018. find
  20. Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637–5664. PMLR, 2… find
  21. Ali Sinan Koksal, Yewen Pu, Saurabh Srivastava, Rastislav Bodik, Jasmin Fisher, and Nir Piterman. Synthesis of biological models from mutation experiments. In Proceedings of the 40th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages, pages 469–482, 2013. find
  22. Tessa A Lau and Daniel S Weld. Programming by demonstration: An inductive learning formulation. In Proceedings of the 4th international conference on Intelligent user interfaces, pages 145–152, 1998. find
  23. Xubo Leng, Margot Wohl, Kenichi Ishii, Pavan Nayak, and Kenta Asahina. Quantifying influence of human choice on the automated detection of Drosophila behavior by a supervised machine learning algorithm. PLoS ONE, 15(12):e0241696, December 2020. ISSN 1932-6203. doi: 10.1371/journal.pone.0241696. — [doi:10.1371/journal.pone.0241696](https://doi.org/10.1371/journal.pone.0241696) find
  24. Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe, Venkatesh N. Murthy, Mackenzie W. Mathis, and Matthias Bethge. Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 2018. find
  25. Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. arXiv preprint arXiv:2111.09259, 2021. — [arXiv:2111.09259](https://arxiv.org/abs/2111.09259) find
  26. Kevin P. Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. find
  27. Talmo D Pereira, Joshua W Shaevitz, and Mala Murthy. Quantifying behavior to understand the brain. Nature neuroscience, 23(12):1537–1549, 2020. find
  28. Talmo D Pereira, Nathaniel Tabris, Arie Matsliah, David M Turner, Junyu Li, Shruthi Ravindranath, Eleni S Papadoyannis, Edna Normand, David S Deutsch, Z. Yan Wang, Grace C McKenzie-Smith, Catalin C Mitelut, Marielisa Diez Castro, John D’Uva, Mikhail Kislin, Dan H Sanes, Sarah D Kocher, Samuel S-H, A… find
  29. Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206–215, 2019.
  30. Nicolae Sapoval, Amirali Aghazadeh, Michael G Nute, Dinler A Antunes, Advait Balaji, Richard Baraniuk, CJ Barberan, Ruth Dannenfelser, Chen Dun, Mohammadamin Edrisi, et al. Current progress and open challenges for applying deep learning across the biosciences. Nature Communications, 13(1):1–12, 2022… find
  31. Cristina Segalin, Jalani Williams, Tomomi Karigo, May Hui, Moriel Zelikowsky, Jennifer J Sun, Pietro Perona, David J Anderson, and Ann Kennedy. The mouse action recognition system (mars) software pipeline for automated analysis of social behaviors in mice. Elife, 10:e63720, 2021. find
  32. Ameesh Shah, Eric Zhan, Jennifer Sun, Abhinav Verma, Yisong Yue, and Swarat Chaudhuri. Learning Differentiable Programs with Admissible Neural Heuristics. In Advances in Neural Information Processing Systems, volume 33, pages 4940–4952. Curran Associates, Inc., 2020. find
  33. Jennifer J Sun, Ann Kennedy, Eric Zhan, David J Anderson, Yisong Yue, and Pietro Perona. Task programming: Learning data efficient behavior representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2876–2885, 2021. find
  34. Megan Tjandrasuwita, Jennifer J Sun, Ann Kennedy, Swarat Chaudhuri, and Yisong Yue. Interpreting expert annotation differences in animal behavior. CV4Animals Workshop at CVPR, 2021. find
  35. Albert Tseng, Jennifer J Sun, and Yisong Yue. Automatic synthesis of diverse weak supervision sources for behavior analysis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. find
  36. Berk Ustun and Cynthia Rudin. Optimized risk scores. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1125–1134, 2017. find
  37. Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. Houdini: Lifelong learning as program synthesis. Advances in Neural Information Processing Systems, 31, 2018. find
  38. Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pages 5045–5054. PMLR, 2018. find
  39. Catherine Wong, Kevin M Ellis, Joshua Tenenbaum, and Jacob Andreas. Leveraging language to learn program abstractions and search heuristics. In International Conference on Machine Learning, pages 11193–11204. PMLR, 2021. find
  40. Eric Zhan, Jennifer J Sun, Ann Kennedy, Yisong Yue, and Swarat Chaudhuri. Unsupervised learning of neurosymbolic encoders. arXiv preprint arXiv:2107.13132, 2021. — [arXiv:2107.13132](https://arxiv.org/abs/2107.13132) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2022-neurosymbolic-programming-science/