Elamur Research⌕
☰
home / papers / 2026 / ArkRepoBench: A Repository-Level Code Completion Benchmark for HarmonyOS Development

ArkRepoBench: A Repository-Level Code Completion Benchmark for HarmonyOS Development

Paper · 2026 · Yanlin Wang, Bowen Zhang, Yanli Wang et al.

Summary

ArkTS, the main language for building HarmonyOS apps, is a low-resource language, and no benchmark has tested how well LLMs complete ArkTS code at the repository level. The authors present ArkRepoBench, which contains 7,519 completion instances drawn from 20 official HarmonyOS application templates on Gitee. They built a custom ArkTS parser to resolve imports and construct a cross-file dependency graph. They then defined three tasks of increasing granularity: random code spans, single API calls, and extended API fragments. Each instance is also labelled by whether external files are absent, present but unused, or actually required. Experiments with open and closed LLMs show that ArkTS completion lags behind Java and Python. Small open models perform about as well as proprietary ones. Retrieved cross-file context helps only when it is truly needed, and lexical retrieval outperforms dense retrieval. Adding official HarmonyOS API documentation improves results reliably. The benchmark gives the HarmonyOS developer community a way to measure LLM coding assistants and points toward better retrieval as a key direction.

Key points

  • Scale and sources: 7,519 samples from 20 Gitee ArkTS templates (API 5.0.0–5.0.4). The tasks are Random Span Completion (4,363), Atomic API Completion (1,593) and Extended API Completion (1,593). The dependency graph has 4,950 nodes and 30,221 edges.
  • Dependency categories: Instances are sorted into three classes: those needing no other files, labelled Standalone (SA); those where related files exist but go unused, labelled Cross-File Independent (CFI); and those that rely on such files, labelled Cross-File Dependent (CFD). These separate the mere presence of related files from a real need for them.
  • Gap to mainstream languages: ArkTS trails Java and Python by 3–7% in Exact Match and 17–22% in Edit Similarity.
  • Open vs. closed models and retrieval: DeepSeekCoder-7B reaches 5.13% repository-level EM, compared with 4.70% for GPT-4o. On CFD cases, retrieval raises 7B EM from 11.80% to 16.36%, and sparse Jaccard retrieval beats dense methods. API documentation adds more than 3% EM on 7B models.
  • Import extraction: ArkAnalyzer captures about 77% of side-effect imports, while the authors' parser captures 100%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🏢Huawei TechnologiesShenzhen · China75 works · 12,975 citations🎓Monash UniversityMelbourne · Australia60 works · 54,866 citations🎓Sun Yat-sen UniversityGuangzhou · China53 works · 9,185 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksCode & programming languages · 129 works

Cite

Cite as Yanlin Wang, Bowen Zhang, Yanli Wang, Daya Guo, Terry Yue Zhuo, Jiachi Chen, Mingwei Liu, Xingong Zhang and Zibin Zheng. 2026. ArkRepoBench: A Repository-Level Code Completion Benchmark for HarmonyOS Development. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.969

@inproceedings{wang2026arkrepobench,
  title = {{ArkRepoBench: A Repository-Level Code Completion Benchmark for HarmonyOS Development}},
  author = {Yanlin Wang, Bowen Zhang, Yanli Wang, Daya Guo, Terry Yue Zhuo, Jiachi Chen, Mingwei Liu, Xingong Zhang and Zibin Zheng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.969},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.969}
}

References

42 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Anthropic. 2025. Introducing Claude 4. https://www.anthropic.com/news/claude-4. Claude Opus 4 and Claude Sonnet 4 released May 22, 2025. ↗
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  4. Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D C, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, Balasubramanyan Ashok, and Shashank Shet. 2024. Codeplan: Repository-level coding using llms and planning. Proceedings of the ACM on Software Engineering, 1(FSE):675–698. find
  5. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, and 1 others. 2023. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software En… find
  6. Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, and 1 others. 2024. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436. — [arXiv:2406.07436](https://arxiv.org/abs/2406.07436) find
  7. Haonan Chen, Daihang Chen, Yizhuo Yang, Lingyun Xu, Liang Gao, Mingyi Zhou, Chunming Hu, and Li Li. 2025. Arkanalyzer: The static analysis framework for openharmony. arXiv preprint arXiv:2501.05798. — [arXiv:2501.05798](https://arxiv.org/abs/2501.05798) find
  8. Mark Chen. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  9. Ken Deng, Jiaheng Liu, He Zhu, Congnan Liu, Jingxin Li, Jiakai Wang, Peng Zhao, Chenchen Zhang, Yanan Wu, Xueqiao Yin, and 1 others. 2024. R2c2-coder: Enhancing and benchmarking real-world repository-level code completion abilities of code large language models. arXiv preprint arXiv:2406.01359. — [arXiv:2406.01359](https://arxiv.org/abs/2406.01359) find
  10. Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and 1 others. 2023. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Syst… find
  11. Yangruibo Ding, Zijian Wang, Wasi Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2024. Cocomic: Code completion by jointly modeling in-file and cross-file context. In Proceedings of the 2024 Joint International Conference on Computational Linguistics,…
  12. Aryaz Eghbali and Michael Pradel. 2024. De-hallucinator: Iterative grounding for llm-based code completion. arXiv preprint arXiv:2401.01701. — [arXiv:2401.01701](https://arxiv.org/abs/2401.01701) find
  13. Google DeepMind. 2025. Gemini 2.5: Our most intelligent AI model with thinking. https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/. Gemini 2.5 Pro released March 25, 2025. ↗
  14. Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. Unixcoder: Unified cross-modal pre-training for code representation. arXiv preprint arXiv:2203.03850.
  15. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501.12948. arXiv preprint Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie…
  16. Huawei Developer. 2024. ArkTS development language. https://developer.huawei.com/ consumer/en/arkts/. ↗
  17. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. — Qwen2.5-Coder Technical Report
  18. Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2024. Xcodeeval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. In Proceedings of the 62nd Annual Meeting of the…
  19. Jia Li, Xuyuan Guo, Lei Li, Kechi Zhang, Ge Li, Zhengwei Tao, Fang Liu, Chongyang Tao, Yuqi Zhu, and Zhi Jin. 2025a. Longcodeu: Benchmarking long-context language models on long code understanding. arXiv preprint arXiv:2503.04359. find
  20. Li Li, Xiang Gao, Hailong Sun, Chunming Hu, Carolyn Sun, Haoyu Wang, Haipeng Cai, Ting Su, Xiapu Luo, Tegawendé Bissyande, and 1 others. 2025b. Software engineering for openharmony: A research roadmap. ACM Computing Surveys, 58(2):1–36. find
  21. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, and 1 others. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161. find
  22. Ming Liang, Xiaoheng Xie, Gehao Zhang, Xunjin Zheng, Peng Di, Hongwei Chen, Chengpeng Wang, Gang Fan, and 1 others. 2024. Repofuse: Repository-level code completion with fused dual context. arXiv preprint arXiv:2402.14323. — [arXiv:2402.14323](https://arxiv.org/abs/2402.14323) find
  23. Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai, Yanan Wu, JinKe JinKe, and 1 others. 2025. M2rc-eval: Massively multilingual repository-level code completion evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Lingui…
  24. Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang. 2024a. Repoqa: Evaluating long context code understanding. arXiv preprint arXiv:2406.06025. — [arXiv:2406.06025](https://arxiv.org/abs/2406.06025) find
  25. Tianyang Liu, Canwen Xu, and Julian McAuley. 2023. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091. find
  26. Wei Liu, Ailun Yu, Daoguang Zan, Bo Shen, Wei Zhang, Haiyan Zhao, Zhi Jin, and Qianxiang Wang. 2024b. Graphcoder: Enhancing repository-level code completion via code context graph-based retrieval and language model. arXiv preprint arXiv:2406.07003. — [arXiv:2406.07003](https://arxiv.org/abs/2406.07003) find
  27. Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, and 1 others. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173. find
  28. Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svyatkovskiy. 2022. Reacc: A retrieval-augmented code completion framework. arXiv preprint arXiv:2203.07722.
  29. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, and 1 others. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664. find
  30. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. find
  31. OpenAI. 2025. GPT-5 system card. https://openai.com/index/introducing-gpt-5/. Released August 7, 2025. ↗
  32. Qiwei Peng, Yekun Chai, and Xuhong Li. 2024. Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization. arXiv preprint arXiv:2402.16694. find
  33. Huy N Phan, Hoang N Phan, Tien N Nguyen, and Nghi DQ Bui. 2025. Repohyper: Search-expand-refine on semantic graphs for repository-level code completion. In 2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge), pages 14–25. IEEE. find
  34. Stephen Robertson, Hugo Zaragoza, and 1 others. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. find
  35. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, and 1 others. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.
  36. Disha Shrivastava, Denis Kocetkov, Harm De Vries, Dzmitry Bahdanau, and Torsten Scholak. 2023. Repofusion: Training code models to understand your repository. arXiv preprint arXiv:2306.10998. find
  37. Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng. 2024. Rlcoder: Reinforcement learning for repository-level code completion. arXiv preprint arXiv:2407.19487. — [arXiv:2407.19487](https://arxiv.org/abs/2407.19487) find
  38. Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. 2024. Repoformer: Selective retrieval for repository-level code completion. arXiv preprint arXiv:2403.10059. — [arXiv:2403.10059](https://arxiv.org/abs/2403.10059) find
  39. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  40. Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570. find
  41. Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint arXiv:2401.07339.
  42. Bo Zhou, Jiaqi Shi, Ying Wang, Li Li, Tsz On Li, Hai Yu, and Zhiliang Zhu. 2025. Porting software libraries to openharmony: Transitioning from typescript or javascript to arkts. Proceedings of the ACM on Software Engineering, 2(ISSTA):1445–1466. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-arkrepobench-repository-level-code-completion/