Elamur Research⌕
☰
home / papers / 2024 / Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code

Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code

Paper · 2024 · Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna et al.

Summary

The paper asks how well large language models can convert programs from one programming language to another, and where they fail. The authors assembled 1,700 runnable samples in C, C++, Go, Java and Python. These came from CodeNet, Avatar and EvalPlus, plus two open-source projects, Apache Commons CLI and Python Click. They ran seven models (GPT-4, Llama 2, TheBloke-Airoboros, TheBloke-Vicuna, StarCoder, CodeGeeX, CodeGen) and judged each output by executing the accompanying tests.

Results were weak overall. Even the strongest model, GPT-4, translated fewer than half of the samples correctly, and the models largely failed on real-world projects. Through manual inspection of failed GPT-4 outputs, the team built a taxonomy of translation bugs. Frequent trouble spots include:

  • choosing matching data types,
  • finding equivalent APIs,
  • replacing language-specific features such as overloading or annotations.

Classic rule-based transpilers showed different strengths, which suggests hybrid tools would work best. An iterative prompting scheme feeds error details back to the model and recovers part of the failures. The released dataset and taxonomy give a solid basis for future research on automated code migration.

Key points

  • Across 43,379 translations, success ranged from 2.1% to 47.3% (mean 11.94%, median 5.3%). GPT-4 led at 47.3%, followed by StarCoder at 14.5%. On the real-world projects, GPT-4 reached 8.1% and all other models 0%.
  • The bug taxonomy has 15 categories in five groups. It was derived from 1,725 faulty GPT-4 translations by eight labelers over 630 person-hours.
  • Non-LLM baselines were CxGo, C2Rust and JavaToCsharp. Their strengths complemented those of the LLMs.
  • Iterative prompts include the failed code, stack traces, error messages and failing test input/output. They raised success by 5.5% on average and by 12% for GPT-4.
  • The public dataset contains:
    • 1,700 samples with 10K+ tests,
    • 43K+ translated programs,
    • 1,748 manually labeled bugs,
    • 1,365 bug-fix pairs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States228 works · 74,591 citations🏢IBMArmonk · United States163 works · 73,139 citations🔬IBM Research - Thomas J. Watson Research CenterYorktown Heights · United States57 works · 49,627 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 worksSoftware · 150 works

Cite

Cite as Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha and Reyhaneh Jabbarvand. 2024. Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code. arXiv:2308.03109.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3597503.3639226 · arXiv 2308.03109

@misc{pan2024lost,
  title = {{Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code}},
  author = {Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha and Reyhaneh Jabbarvand},
  year = {2024},
  doi = {10.1145/3597503.3639226},
  eprint = {2308.03109},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3597503.3639226}
}

References

92 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2018. Upgrading GitHub from Rails 3.2 to 5.2. https://github.blog/2018-09-28-upgrading-github-from-rails-3-2-to-5-2/. ↗
  2. 2020. Supporting Linux kernel development in Rust. https://lwn.net/Articles/ 829858/. ↗
  3. 2020. Transform monolithic Java applications into microservices with the power of AI. https://developer.ibm.com/tutorials/transform-monolithic-java-applications-into-microservices-with-the-power-of-ai/. ↗
  4. 2020. Will code move on to a language such as Rust? https://www.theregister.com/2020/06/30/hard_to_find_linux_maintainers_says_torvalds/. ↗
  5. 2021. GitHub’s Journey from Monolith to Microservices. https://www.infoq.com/ articles/github-monolith-microservices/. ↗
  6. 2023. Apache Commons CLI. https://commons.apache.org/proper/commons-cli/. ↗
  7. 2023. Artifact Website. https://github.com/Intelligent-CAT-Lab/ PLTranslationEmpirical. ↗
  8. 2023. C to Go Translator. https://github.com/gotranspile/cxgo. ↗
  9. 2023. C2Rust Transpiler. https://github.com/immunant/c2rust. ↗
  10. 2023. Click. https://click.palletsprojects.com/en/8.1.x/. ↗
  11. 2023. CodeGeeX. https://github.com/THUDM/CodeGeeX/blob/main/tests/test_-prompt.txt. ↗
  12. 2023. GPT-4 Technical Report. https://cdn.openai.com/papers/gpt-4.pdf. ↗
  13. 2023. Hugging Face Open LLM Leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard. ↗
  14. 2023. Java 2 CSharp Translator for Eclipse. https://sourceforge.net/projects/ j2cstranslator/. ↗
  15. 2023. Java to CSharp Converter. https://github.com/paulirwin/JavaToCSharp. ↗
  16. 2023. Llama-2. https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/. py_compile—Compile Python source files. ↗
  17. 2023. https://docs.python.org/3/library/py_compile.html. ↗
  18. 2023. Sharpen - Automated Java-\>C\# coversion. https://github.com/mono/ sharpen. ↗
  19. 2023. TheBloke Airoboros 13B. https://huggingface.co/TheBloke/airoboros-13B-HF. ↗
  20. 2023. TheBloke Wizard Vicuna 13B. https://huggingface.co/TheBloke/Wizard-Vicuna-13B-Uncensored-HF. ↗
  21. 2023. TIOBE Index. https://www.tiobe.com/tiobe-index/. ↗
  22. Seif Abukhalaf, Mohammad Hamdaqa, and Foutse Khomh. 2023. On Codex Prompt Engineering for OCL Generation: An Empirical Study. arXiv preprint arXiv:2303.16244 (2023). — [arXiv:2303.16244](https://arxiv.org/abs/2303.16244) find
  23. Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-Wei Chang. 2021. Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2108.11590 (2021). find
  24. Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. SantaCoder: don’t reach for the stars! arXiv preprint arXiv:2301.03988 (2023).
  25. Boris Beizer. 1990. Software testing techniques. find
  26. Alexander Bergmayr, Hugo Bruneliere, Javier Luis Cánovas Izquierdo, Jesús Gorronogoitia, George Kousiouris, Dimosthenis Kyriazis, Philip Langer, Andreas Menychtas, Leire Orue-Echevarria, Clara Pezuela, et al. 2013. Migrating legacy software to the cloud with ARTIST. In 2013 17th European Conference… find
  27. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021).
  28. Pinzhen Chen and Gerasimos Lampouras. 2023. Exploring data augmentation for code generation tasks. arXiv preprint arXiv:2302.03499 (2023). find
  29. Xinyun Chen, Chang Liu, and Dawn Song. 2018. Tree-to-tree neural networks for program translation. Advances in neural information processing systems 31 (2018). find
  30. Roberto Rodriguez Echeverria, Fernando Macias, Victor Manuel Pavon, Jose Maria Conejero, and Fernando Sanchez Figueroa. 2015. Legacy web application modernization by generating a REST service layer. IEEE Latin America Transactions 13, 7 (2015), 2379–2383. find
  31. Aryaz Eghbali and Michael Pradel. 2022. CrystalBLEU: precisely and efficiently measuring the similarity of code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–12. find
  32. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020).
  33. L. Gazzola, D. Micucci, and L. Mariani. 2019. Automatic Software Repair: A Survey. IEEE Transactions on Software Engineering 45, 01 (jan 2019), 34–67. https://doi.org/10.1109/TSE.2017.2755013 — [doi:10.1109/tse.2017.2755013](https://doi.org/10.1109/tse.2017.2755013) ↗
  34. Mahdi Fahmideh Gholami, Farhad Daneshgar, Ghassan Beydoun, and Fethi Rabhi. 2017. Challenges in migrating legacy software systems to the cloud—an empirical study. Information Systems 67 (2017), 100–113. find
  35. Linyuan Gong, Jiayi Wang, and Alvin Cheung. 2023. ADELT: Transpilation Between Deep Learning Frameworks. arXiv preprint arXiv:2303.03593 (2023). — [arXiv:2303.03593](https://arxiv.org/abs/2303.03593) find
  36. Sindre Grønstøl Haugeland, Phu H Nguyen, Hui Song, and Franck Chauvel. 2021. Migrating monoliths to microservices-based customizable multi-tenant cloud-native apps. In 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 170–177. find
  37. DongNyeong Heo and Heeyoul Choi. 2022. End-to-End Training of Both Translation Models in the Back-Translation Framework. arXiv preprint arXiv:2202.08465 (2022). — [arXiv:2202.08465](https://arxiv.org/abs/2202.08465) find
  38. Jaemin Hong. 2023. Improving Automatic C-to-Rust Translation with Static Analysis. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 273–277. find
  39. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146 (2018). find
  40. Md Johirul Islam, Giang Nguyen, Rangeet Pan, and Hridesh Rajan. 2019. A comprehensive study on deep learning bug characteristics. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 510–520. find
  41. Md Johirul Islam, Rangeet Pan, Giang Nguyen, and Hridesh Rajan. 2020. Repairing deep neural networks: Fix patterns and challenges. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 1135–1146. find
  42. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural computation 3, 1 (1991), 79–87. find
  43. Prithwish Jana, Piyush Jha, Haoyang Ju, Gautham Kishore, Aryan Mahajan, and Vijay Ganesh. 2023. Attention, Compilation, and Solver-based Symbolic Analysis are All You Need. arXiv preprint arXiv:2306.06755 (2023). — [arXiv:2306.06755](https://arxiv.org/abs/2306.06755) find
  44. Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. 2023. Self-planning code generation with large language model. arXiv preprint arXiv:2303.06689 (2023).
  45. Harshit Joshi, José Cambronero Sanchez, Sumit Gulwani, Vu Le, Gust Verbruggen, and Ivan Radiček. 2023. Repair is nearly generation: Multilingual program repair with llms. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 5131–5140. find
  46. Anup K Kalia, Jin Xiao, Rahul Krishna, Saurabh Sinha, Maja Vukovic, and Debasish Banerjee. 2021. Mono2micro: a practical and effective tool for decomposing monolithic java applications to microservices. In Proceedings of the 29th ACM joint meeting on European software engineering conference and symp… find
  47. Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. 2014. Phrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software. 173–184. find
  48. Justas Kazanavičius and Dalius Mažeika. 2019. Migrating legacy software to microservices architecture. In 2019 Open Conference of Electrical, Electronic and Information Sciences (eStream). IEEE, 1–5. find
  49. Rahul Krishna, Anup Kalia, Saurabh Sinha, Rachel Tzoref-Brill, John Rofrano, and Jin Xiao. 2021. Transforming monolithic applications to microservices with Mono2Micro. In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering. 3–3. find
  50. Marie-Anne Lachaux, Baptiste Roziere, Marc Szafraniec, and Guillaume Lample. 2021. DOBF: A deobfuscation pre-training objective for programming languages. Advances in Neural Information Processing Systems 34 (2021), 14967–14979. find
  51. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. StarCoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023). find
  52. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. StarCoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023). find
  53. Fang Liu, Jia Li, and Li Zhang. 2023. Syntax and Domain Aware Model for Unsupervised Program Translation. arXiv preprint arXiv:2302.03908 (2023). — [arXiv:2302.03908](https://arxiv.org/abs/2302.03908) find
  54. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210 (2023).
  55. Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. 2013. Lexical statistical machine translation for language migration. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. 651–654. find
  56. Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. 2014. Migrating code with statistical machine translation. In Companion Proceedings of the 36th International Conference on Software Engineering. 544–547. find
  57. Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. 2015. Divide-and-conquer approach for multi-phase statistical migration for source code (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 585–596. find
  58. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In The Eleventh International Conference on Learning Representations. find
  59. Vikram Nitin, Shubhi Asthana, Baishakhi Ray, and Rahul Krishna. 2022. CARGO: ai-guided dependency analysis for migrating monolithic applications to microservices architecture. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–12. find
  60. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]
  61. Hongyu Pei Breivold. 2020. Towards factories of the future: migration of industrial legacy automation systems in the cloud computing and Internet-of-things context. Enterprise Information Systems 14, 4 (2020), 542–562. find
  62. Ricardo Pérez-Castillo, Manuel A Serrano, and Mario Piattini. 2021. Software modernization to embrace quantum technology. Advances in Engineering Software 151 (2021), 102933. find
  63. Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. 2021. CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks. In Thirty-fifth Conference on Neural Information Proce…
  64. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297 (2020). find
  65. Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems 33 (2020), 20601–20611.
  66. Baptiste Roziere, Jie M Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2021. Leveraging automated unit tests for unsupervised code translation. arXiv preprint arXiv:2110.06773 (2021).
  67. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 (2022). find
  68. Rajaraajeswari Settu and Pethuru Raj. 2013. Cloud application modernization and migration methodology. Cloud Computing: Methods and Practical Approaches (2013), 243–271. find
  69. André Silva, João F Ferreira, He Ye, and Martin Monperrus. 2023. MUFIN: Improving Neural Repair Models with Back-Translation. arXiv preprint arXiv:2304.02301 (2023). — [arXiv:2304.02301](https://arxiv.org/abs/2304.02301) find
  70. Ishan Mani Subedi, Maninder Singh, Vijayalakshmi Ramasamy, and Gursim-ran Singh Walia. 2021. Application of back-translation: a transfer learning approach to identify ambiguous software requirements. In Proceedings of the 2021 ACM Southeast Conference. 130–137. find
  71. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification?. In Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18. Springer, 194–206.
  72. Qiushi Sun, Nuo Chen, Jianing Wang, Xiang Li, and Ming Gao. 2023. TransCoder: Towards Unified Transferable Code Representation Learning Inspired by Human Skills. arXiv preprint arXiv:2306.07285 (2023). find
  73. Marc Szafraniec, Baptiste Roziere, Hugh Leather Francois Charton, Patrick Labatut, and Gabriel Synnaeve. 2022. Code translation with compiler representations. arXiv preprint arXiv:2207.03578 (2022). — [arXiv:2207.03578](https://arxiv.org/abs/2207.03578) find
  74. Johannes Thönes. 2015. Microservices. IEEE software 32, 1 (2015), 116–116. find
  75. Chih-Kai Ting, Karl Munson, Serenity Wade, Anish Savla, Kiran Kate, and Kavitha Srinivas. 2023. CodeStylist: A System for Performing Code Style Transfer Using Neural Networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 16485–16487. find
  76. Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. 2022. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Chi conference on human factors in computing systems extended abstracts. 1–7. find
  77. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021). find
  78. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837.
  79. Justin D Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021. Perfection not required? Human-AI partnerships in code translation. In 26th International Conference on Intelligent User Interfaces. 402–412. find
  80. Justin D Weisz, Michael Muller, Steven I Ross, Fernando Martinez, Stephanie Houde, Mayank Agarwal, Kartik Talamadupula, and John T Richards. 2022. Better together? an evaluation of ai-supported code translation. In 27th International Conference on Intelligent User Interfaces. 369–391. find
  81. Yuanbo Wen, Qi Guo, Qiang Fu, Xiaqing Li, Jianxing Xu, Yanlin Tang, Yongwei Zhao, Xing Hu, Zidong Du, Ling Li, et al. 2022. BabelTower: Learning to Auto-parallelized Program Translation. In International Conference on Machine Learning. PMLR, 23685–23700. find
  82. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023).
  83. Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2022. Practical program repair in the era of large pre-trained language models. arXiv preprint arXiv:2210.14179 (2022). — [arXiv:2210.14179](https://arxiv.org/abs/2210.14179) find
  84. Chunqiu Steven Xia and Lingming Zhang. 2022. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 959–971. find
  85. Chunqiu Steven Xia and Lingming Zhang. 2023. Conversational automated program repair. arXiv preprint arXiv:2301.13246 (2023). — [arXiv:2301.13246](https://arxiv.org/abs/2301.13246) find
  86. Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming. 1–10.
  87. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601 (2023). find
  88. Quanjun Zhang, Chunrong Fang, Yuxiang Ma, Weisong Sun, and Zhenyu Chen. 2023. A Survey of Learning-based Automated Program Repair. arXiv preprint arXiv:2301.03270 (2023). — [arXiv:2301.03270](https://arxiv.org/abs/2301.03270) find
  89. WG Zhang, Arne J Berre, Dumitru Roman, and Hans Aage Huru. 2009. Migrating legacy applications to the service Cloud. In Proceedings of the 14th Conference Companion on Object Oriented Programming Systems Languages and Applications. 59–68. find
  90. Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, and Lu Zhang. 2018. An empirical study on TensorFlow program bugs. In Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis. 129–140. find
  91. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. 2023. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568 (2023).
  92. Terry Yue Zhuo, Zhuang Li, Yujin Huang, Yuan-Fang Li, Weiqing Wang, Gholamreza Haffari, and Fatemeh Shiri. 2023. On robustness of prompt-based semantic parsing with large pre-trained language model: An empirical study on codex. arXiv preprint arXiv:2301.12868 (2023). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pan-2024-lost-translation-study-bugs-introduced/