Elamur Research⌕
☰
home / papers / 2021 / Self-Supervised Bug Detection and Repair

Self-Supervised Bug Detection and Repair

Paper · 2021 · Miltiadis Allamanis, Henry Jackson-Flux and Marc Brockschmidt

Summary

The authors study how to train a neural network to find and fix bugs in code when no large labeled datasets of real bugs are available. They propose BugLab, a self-supervised approach in which two models are trained jointly and compete with each other. The selector model chooses which change to make to correct code so as to create the bug that is hardest to detect. The detector model searches for the location of the introduced bug and proposes the reverse fix. Both models formulate the task in the same way, as rewriting the syntax tree, and represent code as a graph of entities and relations: syntax, data and control flow, and types. Representations are computed with a GNN or the relational transformer GREAT. The Python implementation, PyBugLab, focuses on small bugs such as typos or copy-paste errors. On the new dataset of real bugs, PyPIBugs, it outperforms baseline methods trained on randomly inserted bugs, and it found previously unknown bugs in open-source packages. At the same time, the work shows that performance on synthetic and on real bugs differs substantially and that the false positive rate is still too high for practical use.

Key points

  • PyBugLab covers four bug classes: variable misuse, wrong operator, wrong literal, and swapped call arguments.
  • PyPIBugs is a new test set of 2374 real bugs, selected manually. The improvement over baseline methods reaches 30%.
  • When checking popular packages from PyPI, only 19 of 1000 warnings turned out to be real bugs, i.e., about 98% of the alerts are false positives. The authors reported 11 of these bugs on GitHub, and 6 fixes have already been accepted.
  • The idea is close to ELECTRA and GANs. The difference is that the objective is non-differentiable, and after training the selector is discarded and only the detector is used to find bugs.
  • The code and PyPIBugs are published in the microsoft/neurips21-self-supervised-bug-detection-and-repair repository on GitHub. The paper appeared at NeurIPS 2021.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States404 works · 163,863 citations🏢Microsoft Research (United Kingdom)Cambridge · United Kingdom15 works · 3,537 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksInformation Systems · 1,888 worksSoftware · 150 works

Cite

Cite as Miltiadis Allamanis, Henry Jackson-Flux and Marc Brockschmidt. 2021. Self-Supervised Bug Detection and Repair. arXiv:2105.12787.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2105.12787 · arXiv 2105.12787

@misc{allamanis2021self,
  title = {{Self-Supervised Bug Detection and Repair}},
  author = {Miltiadis Allamanis, Henry Jackson-Flux and Marc Brockschmidt},
  year = {2021},
  doi = {10.48550/arxiv.2105.12787},
  eprint = {2105.12787},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2105.12787}
}

References

33 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Allamanis. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, pages 143–153, 2019.
  2. M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), 51(4):1–37, 2018.
  3. M. Allamanis, M. Brockschmidt, and M. Khademi. Learning to represent programs with graphs. In International Conference on Learning Representations (ICLR), 2018.
  4. M. Allamanis, E. Barr, S. Ducousso, and Z. Gao. Typilus: Neural type hints. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 91–105. Association for Computing Machinery (ACM), 2020. find
  5. T. Bachlechner, B. P. Majumder, H. H. Mao, G. W. Cottrell, and J. McAuley. Rezero is all you need: Fast convergence at large depth. arXiv preprint arXiv:2003.04887, 2020.
  6. K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020. find
  7. M. Cvitkovic, B. Singh, and A. Anandkumar. Deep learning on code with an unbounded vocabulary. In Machine Learning 4 Programming, 2018. find
  8. M. Dennis, N. Jaques, E. Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. arXiv preprint arXiv:2012.02096, 2020. find
  9. E. Dinella, H. Dai, Z. Li, M. Naik, L. Song, and K. Wang. Hoppity: Learning graph transformations to detect and fix bugs in programs. In International Conference on Learning Representations, 2019. find
  10. J. Eberhardt, S. Steffen, V. Raychev, and M. Vechev. Unsupervised learning of API aliasing specifications. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 745–759, 2019. find
  11. J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pages 1263–1272, 2017.
  12. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661, 2014.
  13. V. J. Hellendoorn, C. Sutton, R. Singh, P. Maniatis, and D. Bieber. Global relational models of source code. In International Conference on Learning Representations, 2020. find
  14. R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 654–665, 2014. find
  15. A. Kanade, P. Maniatis, G. Balakrishnan, and K. Shi. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning (ICML), 2020. find
  16. R.-M. Karampatsis and C. Sutton. How often do single-statement bugs occur? the ManySStuBs4J dataset. In Proceedings of the 17th International Conference on Mining Software Repositories, pages 573–577, 2020. find
  17. J. Katz. Libraries.io Open Source Repository and Dependency Metadata, Jan. 2020. URL https://doi.org/10.5281/zenodo.3626071. — [doi:10.5281/zenodo.3626071](https://doi.org/10.5281/zenodo.3626071) ↗
  18. Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel. Gated graph sequence neural networks. In International Conference on Learning Representations (ICLR), 2016. find
  19. S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. In International Conference on Learning Representations (ICLR), 2017.
  20. A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in PyTorch. 2017. find
  21. J. Patra and M. Pradel. Semantic bug seeding: a learning-based approach for creating realistic bugs. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 906–918, 2021. find
  22. M. Pradel and K. Sen. DeepBugs: A learning approach to name-based bug detection. Proceedings of the ACM on Programming Languages, 2(OOPSLA):1–25, 2018. find
  23. G. Ramakrishnan, J. Henkel, Z. Wang, A. Albarghouthi, S. Jha, and T. Reps. Semantic robustness of models of source code. arXiv preprint arXiv:2002.03043, 2020. — [arXiv:2002.03043](https://arxiv.org/abs/2002.03043) find
  24. B. Ray, V. Hellendoorn, S. Godhane, Z. Tu, A. Bacchelli, and P. Devanbu. On the ”naturalness” of buggy code. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pages 428–439. IEEE, 2016. find
  25. V. Raychev, M. Vechev, and A. Krause. Predicting program properties from Big Code. In Principles of Programming Languages (POPL), 2015. find
  26. A. Rice, E. Aftandilian, C. Jaspan, E. Johnston, M. Pradel, and Y. Arroyo-Paredes. Detecting argument selection defects. Proceedings of the ACM on Programming Languages, 1(OOPSLA): 1–22, 2017. find
  27. D. Spadini, M. Aniche, and A. Bacchelli. PyDriller: Python framework for mining software repositories. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering - ESEC/FSE 2018, pages 908–911, New York, New… — [doi:10.1145/3236024.3264598](https://doi.org/10.1145/3236024.3264598) find
  28. D. Tarlow, S. Moitra, A. Rice, Z. Chen, P.-A. Manzagol, C. Sutton, and E. Aftandilian. Learning to fix build errors with Graph2Diff neural networks. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, pages 19–20, 2020. find
  29. H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103.17239, 2021. find
  30. M. Vasic, A. Kanade, P. Maniatis, D. Bieber, and R. Singh. Neural program repair by jointly learning to localize and repair. arXiv preprint arXiv:1904.01720, 2019. — [arXiv:1904.01720](https://arxiv.org/abs/1904.01720) find
  31. K. Wang and Z. Su. Blended, precise semantic program embeddings. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 121–134, 2020. find
  32. S. Wang, D. Chollak, D. Movshovitz-Attias, and L. Tan. Bugram: bug detection with n-gram language models. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, pages 708–719, 2016. find
  33. J. Wei, M. Goyal, G. Durrett, and I. Dillig. LambdaNet: Probabilistic type inference using graph neural networks. arXiv preprint arXiv:2005.02161, 2020. — [arXiv:2005.02161](https://arxiv.org/abs/2005.02161) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/allamanis-2021-self-supervised-bug-detection-repair/