Elamur Research⌕
☰
home / papers / 2017 / A parallel corpus of Python functions and documentation strings for automated code documentation and code generation

A parallel corpus of Python functions and documentation strings for automated code documentation and code generation

Paper · 2017 · Antonio Valerio Miceli Barone and Rico Sennrich

Summary

Research on translating between source code and natural language has been limited by training data that is tiny, tied to one domain, or noisy. Low-noise sets such as DJANGO and Hearthstone also appear unrealistically easy because they rely on pseudo-code or highly repetitive code. Miceli Barone and Sennrich address this by mining Python 2.7 repositories on GitHub and pairing each top-level function with the docstring its own author wrote. Each function is split into a declaration, a docstring and a body. The code is normalized by parsing and unparsing, and each example is tagged with provenance metadata so that project-level context can be reconstructed. Functions without docstrings go into a separate code-only corpus. As baselines, the authors train Nematus NMT models with BPE in both directions. They also augment code generation through back-translation with synthetic docstrings. The scores are low, at roughly 10–14 BLEU, compared with the 60–85 reported on earlier corpora. This suggests the new resource better reflects real-world difficulty and offers a demanding benchmark.

Key points

  • The main parallel corpus holds 150,370 declaration–docstring–body triples. After duplicate removal it is split into 109,108 training, 2,000 validation and 2,000 test examples. A code-only corpus adds 161,630 functions.
  • Every example records where it came from: the owner and name of its repository, the file it appears in, and its line number.
  • The baselines use Nematus with Moses tokenization and BPE (vocabulary of 89,500), the Adam optimizer, and Bayesian recurrent dropout. No knowledge of program syntax is used.
  • BLEU results (validation/test):
    • Code-to-docstring: 14.03/13.84.
    • Docstring-to-code: 10.32/10.24.
    • Docstring-to-code with back-translation: 10.85/10.90, a gain of about 0.5–0.6 points.
  • The data, extraction scripts, system configurations and model outputs are released at github.com/Avmb/code-docstring-corpus.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

🧭 Topics 1Research subjects and languages the work is about.Code & programming languages · 194 works

Cite

Cite as Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A parallel corpus of Python functions and documentation strings for automated code documentation and code generation. arXiv:1707.02275.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1707.02275 · arXiv 1707.02275

@misc{barone2017parallel,
  title = {{A parallel corpus of Python functions and documentation strings for automated code documentation and code generation}},
  author = {Antonio Valerio Miceli Barone and Rico Sennrich},
  year = {2017},
  doi = {10.48550/arxiv.1707.02275},
  eprint = {1707.02275},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1707.02275}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. 2015a. Suggesting accurate method and class names. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pages 38–49. ACM. find
  2. Miltos Allamanis and Charles Sutton. 2013. Mining source code repositories at massive scale using language modeling. In Working Conference on Mining Software Repositories (MSR). find
  3. Miltos Allamanis, Daniel Tarlow, Andrew Gordon, and Yi Wei. 2015b. Bimodal modelling of source code and natural language. In Proceedings of the 32nd International Conference on Machine Learning, pages 2123–2132, Lille, France. PMLR.
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  5. Avishkar Bhoopchand, Tim Rocktäschel, Earl Barr, and Sebastian Riedel. 2016. Learning python code suggestion with a sparse pointer network. arXiv preprint arXiv:1611.08307. find
  6. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. 2016. Report on the 13th IWSLT Evaluation Campaign. In IWSLT 2016, Seattle, USA. find
  7. Noam Chomsky. 1956. Three models for the description of language. IRE Transactions on information theory, 2(3):113–124. find
  8. Yarin Gal and Zoubin Ghahramani. 2016. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. In Advances in Neural Information Processing Systems 29 (NIPS).
  9. Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. arXiv preprint arXiv:1410.5401. find
  10. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 2073–2083. find
  11. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In The International Conference on Learning Representations, San Diego, California, USA.
  12. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interacti… find
  13. Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, Andrew Senior, Fumin Wang, and Phil Blunsom. 2016. Latent predictor networks for code generation. arXiv preprint arXiv:1603.06744.
  14. George A Miller. 2003. The cognitive revolution: a historical perspective. Trends in cognitive sciences, 7(3):141–144. find
  15. Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Learning to generate pseudo-code from source code using statistical machine translation (t). In Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on, pag… find
  16. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
  17. Chris Quirk, Raymond Mooney, and Michel Galley. 2015. Language to code: Learning semantic parsers for if-this-then-that recipes. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages…
  18. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a Toolkit for Neural Machine Translation. In Proceedings of the Demonstrations at the 15th…
  19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 86–96, Berlin, Germany. find
  20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725, Berlin, Germany.
  21. Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In The 55th Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver, Canada.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/barone-2017-parallel-corpus-python-functions-documentation/