Elamur Research⌕
☰
home / papers / 2020 / PyMT5: multi-mode translation of natural language and Python code with transformers

PyMT5: multi-mode translation of natural language and Python code with transformers

Paper · 2020 · Colin Clement, Dawn Drain, Jonathan Timcheck et al.

Summary

The paper tackles joint modeling of Python source code and its natural-language documentation. The authors propose a single sequence-to-sequence transformer, PyMT5, that converts in both directions: it writes method bodies from signatures or docstrings, and it summarizes code into docstrings. Code and text are handled as ordinary token sequences with a shared BPE vocabulary, rather than as separate modalities with syntax-tree features. The training data comes from about 118k popular GitHub Python repositories, yielding 26 million methods, of which 7.7 million have docstrings. The model is first pre-trained with a T5-style span-masking denoising objective on raw files. It is then fine-tuned in a multi-mode setup: every combination of signature, docstring and body can serve as input or output, provided the same feature is not on both sides. Comment-style prefixes select the target, including the docstring style. PyMT5 beats similarly sized GPT2 baselines, whether randomly initialized or English-pretrained. This matters for practical tools such as documentation generation and code completion.

Key points

  • Corpus: 26M Python methods and 7.7M method–docstring pairs. These come from 2.3M deduplicated files (about 27GB of raw text). The authors call it the largest such parallel corpus reported, over 15× the next largest Python one in CodeSearchNet (CSN).
  • Model: encoder–decoder transformer with 374M parameters and a 50,181-token BPE vocabulary. It was pre-trained for 3 weeks on 16 Tesla V100 GPUs.
  • Effect of pre-training: docstring generation converged about 25× faster, and to a lower loss, than training from random initialization.
  • Results on the CSN test set:
    • 92.1% of predicted method bodies are syntactically correct.
    • Method generation reaches BLEU 8.59 and ROUGE-L F 24.8.
    • Docstring generation reaches BLEU 16.3 and ROUGE-L F 36.7.
  • Data cleaning for fine-tuning: methods were extracted and re-generated through Python's ast and astunparse. As a result, the model never sees syntactically invalid code during fine-tuning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations🏢MicrosoftRedmond · United States378 works · 161,630 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

Cite

Cite as Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy and Neel Sundaresan. 2020. PyMT5: multi-mode translation of natural language and Python code with transformers. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.728 · arXiv 2010.03150

@inproceedings{clement2020pymt5,
  title = {{PyMT5: multi-mode translation of natural language and Python code with transformers}},
  author = {Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy and Neel Sundaresan},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.728},
  eprint = {2010.03150},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.728}
}

References

36 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2011. Java doc. Technical report. find
  2. Miltiadis Allamanis, Daniel Tarlow, Andrew D. Gordon, and Yi Wei. 2015. Bimodal modelling of source code and natural language. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, page 2123–2132. JMLR.org. find
  3. Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2018. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400.
  4. Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A parallel corpus of python functions and documentation strings for automated code documentation and code generation. arXiv preprint arXiv:1707.02275. find
  5. Marcel Bruch, Martin Monperrus, and Mira Mezini. 2009. Learning from examples to improve code completion systems. In Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering, pages 213–222. find
  6. Christopher Bryant, Mariano Felice, and Edward Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. Association for Computational Linguistics.
  7. Paweł Budzianowski and Ivan Vulić. 2019. Hello, it’s gpt-2–how can i help you? towards the use of pretrained language models for task-oriented dialogue systems. arXiv preprint arXiv:1907.05774. find
  8. Adelina Ciurumelea, Sebastian Proksch, and Harald Gall. 2020. Suggesting comment completions for python using neural language models. In 27th edition of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE. find
  9. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  11. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint arXiv:1909.00512. find
  12. David Goodger and Guido van Rossum. 2001. Docstring conventions. PEP 257. find
  13. Google. 2020. Google python style guide. Technical report. find
  14. Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018. Deep code search. In Proceedings of the 40th International Conference on Software Engineering, ICSE ’18, page 933–944, New York, NY, USA. Association for Computing Machinery. find
  15. Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. 2012. On the naturalness of software. In 2012 34th International Conference on Software Engineering (ICSE), pages 837–847. IEEE. find
  16. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436.
  17. Richard Jones. 2013. A restructuredtext primer. docutils. sourceforge. net, March. find
  18. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of bert. arXiv preprint arXiv:1908.08593.
  19. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
  20. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345.
  21. Numpydoc Maintainers. 2020. Numpydoc docstring guide. Technical report. find
  22. Laura Moreno, Jairo Aponte, Giriprasad Sridhara, Andrian Marcus, Lori Pollock, and K Vijay-Shanker. 2013. Automatic generation of natural language summaries for java classes. In 2013 21st International Conference on Program Comprehension (ICPC), pages 23–32. IEEE. find
  23. Laura Moreno, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrian Marcus, and Gerardo Canfora. 2014. Automatic generation of release notes. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 484–495. find
  24. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf. ↗
  25. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
  26. Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 419–428. find
  27. Simone Scalabrino, Gabriele Bavota, Christopher Vendome, Mario Linares-Vásquez, Denys Poshyvanyk, and Rocco Oliveto. 2017. Automatically assessing code understandability: How far are we? In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 417–427. IEEE. gen… find
  28. Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. Intellicode compose: Code generation using transformer. arXiv preprint arXiv:2005.08025. find
  29. Juan Zhai, Xiangzhe Xu, Yu Shi, Minxue Pan, Shiqing Ma, Lei Xu, Weifeng Zhang, Lin Tan, and Xiangyu Zhang. 2019. Cpc: automatically classifying and propagating natural language comments via program analysis. find
  30. Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, and Neel Sundaresan. 2019. Pythia: Ai-assisted code completion system. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2727–2735. find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998– 6008.
  32. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. arXiv preprint arXiv:1909.01380.
  33. Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S Yu. 2018. Improving automatic source code summarization via deep reinforcement learning. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pages 397–407. find
  34. Wenhua Wang, Yuqun Zhang, Zhengran Zeng, and Guandong Xu. 2020. Transˆ 3: A transformer-based framework for unifying code summarization and code search. arXiv preprint arXiv:2003.03238. — [arXiv:2003.03238](https://arxiv.org/abs/2003.03238) find
  35. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-ofthe-art natural language processing. ArXiv, abs/1910.03771.
  36. Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/clement-2020-pymt5-multi-mode-translation-natural/