Elamur Research⌕
☰
home / papers / 2026 / What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations

What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations

Paper · 2026 · Yujie Luo, Zhuoyun Yu, Xuehai Wang et al.

Summary

LLM agents that try to reproduce AI papers often fail to produce working code. Papers leave out implementation specifics, and standard RAG does not recover technical details buried in cited works or the code signals in related repositories. The authors propose Executable Knowledge Graphs (xKG), a plug-in knowledge base organized around individual papers. It has a hierarchy of Paper, Technique and Code nodes. Structural edges connect techniques, and implementation edges tie each technique to runnable code, which includes an implementation, a test script and documentation.

Construction is fully automated:

  • o4-mini identifies the core techniques of each target paper.
  • The system gathers related papers that have official GitHub repositories.
  • Code snippets are retrieved for each technique and repaired through a self-debugging loop.
  • Any technique that cannot be grounded in code is discarded.

At replication time, agents use the graph for planning (paper overviews) and for implementation (technique–code pairs), with an LLM verifier screening retrieved items. On PaperBench Code-Dev, xKG improves three agent frameworks with both backbones tested. This suggests that structured, executable knowledge is a general way to strengthen automated research replication.

Key points

  • The graph was built from 42 curated papers (591,145 tokens), using o4-mini and text-embedding-3-small. Repositories on PaperBench's blacklist were excluded to avoid leakage.
  • xKG was evaluated with BasicAgent, IterativeAgent and PaperCoder on o3-mini and DeepSeek-R1, using the PaperBench Code-Dev lite subset and best@3 Replication Score.
    • The largest average gain was +10.90 points for PaperCoder with o3-mini (42.31 → 53.21).
    • IterativeAgent with DeepSeek-R1 gained +8.20.
  • Gains vary strongly by paper. For BasicAgent with o3-mini, MU-DPO rose by 24.26 points, while FRE fell slightly (−0.15).
  • In the ablation study, removing Code Nodes caused the largest drop (−4.56), compared with −2.13 for Paper Nodes and −1.05 for Technique Nodes.
  • Human evaluation of graph quality found:
    • 89.44% of techniques are valid self-contained concepts.
    • 74.51% of technique–code pairs match exactly.
    • All Code Nodes are executable, up from 52.38% before self-debugging.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations🏢Ant GroupHangzhou · China51 works · 1,211 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksInformation Systems · 1,730 worksDecision Sciences · 249 worksInformation Systems and Management · 52 works

🧭 Topics 1Research subjects and languages the work is about.Knowledge graphs & ontologies · 211 works

Cite

Cite as Yujie Luo, Zhuoyun Yu, Xuehai Wang, Yuqi Zhu, Ningyu Zhang, Lanning Wei, Lun Du, Da Zheng and Huajun Chen. 2026. What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.70

@inproceedings{luo2026what,
  title = {{What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations}},
  author = {Yujie Luo, Zhuoyun Yu, Xuehai Wang, Yuqi Zhu, Ningyu Zhang, Lanning Wei, Lun Du, Da Zheng and Huajun Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.70},
  url = {https://doi.org/10.18653/v1/2026.acl-short.70}
}

References

28 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. This work has several limitations. First, the Paper-Bench task exhibits high variance and is costly to evaluate. Although we report results across multiple papers and conduct experiments, due to funding constraints, we only perform experiments on the lite collection of PaperBench Code-Dev. Second, f… find
  2. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. 2024. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095.
  3. Qiguang Chen, Mingda Yang, Libo Qin, Jinhao Liu, Zheng Yan, Jiannan Guan, Dengyun Peng, Yiyan Ji, Hanjing Li, Mengkang Hu, et al. 2025. Ai4research: A survey of artificial intelligence for scientific research. arXiv preprint arXiv:2507.01903.
  4. Nicholas Edwards, Yukyung Lee, Yujun Audrey Mao, Yulu Qin, Sebastian Schuster, and Najoung Kim. 2025. Rexbench: Can coding agents autonomously implement ai research extensions? arXiv preprint arXiv:2506.22598. find
  5. Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. 2024. Unsupervised zero-shot reinforcement learning via functional reward encodings. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. Open-Review.net. find
  6. Manuel Glöckler, Michael Deistler, Christian Dietrich Weilbach, Frank Wood, and Jakob H. Macke. 2024. All-in-one simulation-based inference. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. Open-Review.net. find
  7. Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Huichi Zhou, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, et al. 2025. Deep research agents: A systematic examination and roadmap. arXiv preprint arXiv:2506.18096. — [arXiv:2506.18096](https://arxiv.org/abs/2506.18096) find
  8. Hangyi Jia, Yuxi Qian, Hanwen Tong, Xinhui Wu, Lin Chen, and Feng Wei. 2025. Towards adaptive ml benchmarks: Web-agent-driven construction, domain expansion, and metric optimization. arXiv preprint arXiv:2509.09321. — [arXiv:2509.09321](https://arxiv.org/abs/2509.09321) find
  9. Patrick Tser Jern Kon, Jiachen Liu, Xinyi Zhu, Qiuyi Ding, Jingjia Peng, Jiarong Xing, Yibo Huang, Yiming Qiu, Jayanth Srinivasa, Myungjin Lee, et al. 2025. Exp-bench: Can ai conduct ai research experiments? arXiv preprint arXiv:2505.24785. find
  10. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. 2024. A mechanistic understanding of alignment algorithms: A case study on DPO and toxicity. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024.…
  11. Jiacheng Miao, Joe R Davis, Jonathan K Pritchard, and James Zou. 2025. Paper2agent: Reimagining research papers as interactive and reliable ai agents. arXiv preprint arXiv:2509.06917. — [arXiv:2509.06917](https://arxiv.org/abs/2509.06917) find
  12. Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, et al. 2025. Mlgym: A new framework and benchmark for advancing ai research agents. arXiv preprint arXiv:2502.14499. find
  13. Xuanle Zhao, Zilin Sang, Yuxuan Li, Qi Shi, Weilun Zhao, Shuo Wang, Duzhen Zhang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2025. Autoreproduce: Automatic ai experiment reproduction with paper lineage. arXiv preprint arXiv:2505.20662. find
  14. Shuaicheng Niu, Chunyan Miao, Guohao Chen, Pengcheng Wu, and Peilin Zhao. 2024. Test-time model adaptation with only forward passes. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. Open-Review.net. find
  15. Yixin Ou, Yujie Luo, Jingsheng Zheng, Lanning Wei, Shuofei Qiao, Jintian Zhang, Da Zheng, Huajun Chen, and Ningyu Zhang. 2025. Automind: Adaptive knowledgeable agent for automated data science. arXiv preprint arXiv:2506.10974. — [arXiv:2506.10974](https://arxiv.org/abs/2506.10974) find
  16. Guillaume Sanchez, Alexander Spangher, Honglu Fan, Elad Levi, and Stella Biderman. 2024. Stay on topic with classifier-free guidance. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. find
  17. Zhuoxun Zheng, Baifan Zhou, Dongzhuoran Zhou, Ahmet Soylu, and Evgeny Kharlamov. 2022a. Executable knowledge graph for transparent machine learning in welding monitoring at bosch. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 5102–5103. find
  18. Zhuoxun Zheng, Baifan Zhou, Dongzhuoran Zhou, Ahmet Soylu, and Evgeny Kharlamov. 2022b. Exekg: Executable knowledge graph system for user-friendly data analytics. In Proceedings of the 31st ACM international conference on information & knowledge management, pages 5064–5068. find
  19. Dongzhuoran Zhou, Baifan Zhou, Zhuoxun Zheng, Zhipeng Tan, Egor V Kostylev, and Evgeny Kharlamov. 2022. Towards executable knowledge graph translation. In ISWC (Posters/Demos/Industry). find
  20. Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. 2025. Paper2code: Automating code generation from scientific papers in machine learning. arXiv preprint arXiv:2504.17192. find
  21. Mingyang Zhou, Quanming Yao, Lun Du, Lanning Wei, and Da Zheng. 2025. Reflective paper-to-code reproduction enabled by fine-grained verification. arXiv preprint arXiv:2508.16671. — [arXiv:2508.16671](https://arxiv.org/abs/2508.16671) find
  22. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. 2025. Paperbench: Evaluating ai’s ability to replicate ai research. arXiv preprint arXiv:2504.01848.
  23. Minjun Zhu, Qiujie Xie, Yixuan Weng, Jian Wu, Zhen Lin, Linyi Yang, and Yue Zhang. 2025. Ai scientists fail without strong implementation capability. arXiv preprint arXiv:2506.01372. find
  24. Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. 2025. Ai-researcher: Autonomous scientific innovation. CoRR, abs/2505.18705. find
  25. Edan Toledo, Karen Hambardzumyan, Martin Josifoski, Rishi Hazra, Nicolas Baldwin, Alexis Audran-Reiss, Michael Kuchnik, Despoina Magka, Minqi Jiang, Alisia Maria Lupidi, et al. 2025. Ai research agents for machine learning: Search, exploration, and generalization in mle-bench. arXiv preprint arXiv:2… — [arXiv:2507.02554](https://arxiv.org/abs/2507.02554) find
  26. Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. 2024. How easily do irrelevant inputs skew the responses of large language models? CoRR, abs/2404.03302. find
  27. Shuo Yan, Ruochen Li, Ziming Luo, Zimu Wang, Daoyang Li, Liqiang Jing, Kaiyu He, Peilin Wu, George Michalopoulos, Yue Zhang, et al. 2025. Lmr-bench: Evaluating llm agent’s ability on reproducing language modeling research. arXiv preprint arXiv:2506.17335. find
  28. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2026-what-makes-ai-research-replicable/