Elamur Research⌕
☰
home / papers / 2026 / Lunar-Bench: Towards Evaluating Task-Oriented Reasoning of LLMs in Lunar Exploration Scenarios

Lunar-Bench: Towards Evaluating Task-Oriented Reasoning of LLMs in Lunar Exploration Scenarios

Paper · 2026 · Xin-Yu Xiao, Ye Tian, Erwei Yin et al.

Summary

Large language models are increasingly proposed as assistants for lunar missions, but standard benchmarks test isolated, static questions. They do not show whether a model can reason through interdependent steps under physical, safety and resource limits. The authors build Lunar-Bench, a set of 3,000 tasks derived from mission logs, manuals and technical documents from NASA, ESA, CNSA and other agencies, supplemented by scientific literature. Raw material was standardized, filtered for relevance with Qwen-2.5-72B, and checked by experts. Perceptual factors such as illumination are encoded as structured text. The authors frame the tasks as sequential decision-making and score them with the Environmental Scenario Indicators (ESI), which measure safety, efficiency, integrity and alignment in addition to accuracy. Across 36 models, closed-source systems beat open-source ones. Human experts achieve 65.1% accuracy, while even the top-performing model manages just 47.8%. Chain-of-thought prompting and larger model size bring little improvement. The benchmark highlights how far current LLMs are from dependable mission-level autonomy and offers a domain-specific tool for measuring progress.

Key points

  • Lunar-Bench contains 3,000 tasks covering navigation, energy management, sample analysis and communication coordination. The source data come from space-agency documentation and peer-reviewed research.
  • ESI is a process-based metric with four dimensions: safety (respecting constraints), efficiency (reasoning depth relative to compute cost), integrity (factual grounding of the reasoning trace) and alignment (stability under changing conditions).
  • Of the 36 evaluated LLMs, the best scores 47.8% accuracy, versus 65.1% for human experts. CoT yields only limited gains.
  • In error analysis on a rover data-packet transmission task, common failures included:
    • overlooking details, such as treating lossy compression as lossless;
    • calculation and unit mistakes;
    • outputs truncated near token limits;
    • unwarranted refusals;
    • answers in the wrong format.
  • Data construction combined real protocols with LLM-assisted filtering and augmentation, validated by expert review (IAA = 0.87). The authors acknowledge two limitations: the risk of synthetic bias and the absence of embodied, multimodal evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓Sun Yat-sen UniversityGuangzhou · China72 works · 9,502 citations🎓University of Chinese Academy of SciencesBeijing · China188 works · 8,744 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 works

Cite

Cite as Xin-Yu Xiao, Ye Tian, Erwei Yin, Zhixian He, Shiqi Wang, Yalei Liu and Qianchen Xia. 2026. Lunar-Bench: Towards Evaluating Task-Oriented Reasoning of LLMs in Lunar Exploration Scenarios. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.83

@inproceedings{xiao2026lunar,
  title = {{Lunar-Bench: Towards Evaluating Task-Oriented Reasoning of LLMs in Lunar Exploration Scenarios}},
  author = {Xin-Yu Xiao, Ye Tian, Erwei Yin, Zhixian He, Shiqi Wang, Yalei Liu and Qianchen Xia},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.83},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.83}
}

References

97 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. China National Space Administration. 2025. Official website of cnsa. https://www.cnsa.gov.cn/ english/. Accessed: 2025-05-13. ↗
  3. Jagriti Agrawal, Amruta Yelamanchili, and Steve Chien. 2020. Using explainable scheduling for the mars 2020 rover mission. arXiv preprint arXiv:2011.08733. — [arXiv:2011.08733](https://arxiv.org/abs/2011.08733) find
  4. Huthaifa I Ashqar. 2025. A critical review of benchmarking llms for real-world applications: Trends and limitations. In 2025 Sixteenth International Conference on Ubiquitous and Future Networks (ICUFN), pages 344–346. IEEE. find
  5. Bishwaranjan Bhattacharjee, Aashka Trivedi, Masayasu Muraoka, Muthukumaran Ramasubramanian, Takuma Udagawa, Iksha Gurung, Nishan Pantha, Rong Zhang, Bharath Dandala, Rahul Ramachandran, and 1 others. 2024. Indus: Effective and efficient language models for scientific applications. arXiv preprint arX… find
  6. Joe Burt and Bob Smith. 2012. Deep space climate observatory: The dscovr mission. In 2012 ieee aerospace conference, pages 1–13. IEEE. find
  7. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, and 1 others. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1–45.
  8. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397. find
  9. Shuhao Chen, Han Wang, Mingyu Zhao, Lin Xu, Kai Yu, and Wei Li. 2024a. Routerdc: Query-based router by dual contrastive learning for assembling large language models. In Advances in Neural Information Processing Systems, volume 37, pages 66305–66328. find
  10. Yuanpei Chen, Chen Wang, Li Fei-Fei, and C Karen Liu. 2023. Sequential dexterity: Chaining dexterous policies for long-horizon manipulation. arXiv preprint arXiv:2309.00987. — [arXiv:2309.00987](https://arxiv.org/abs/2309.00987) find
  11. Zhaorun Chen, Zhuokai Zhao, Zhihong Zhu, Ruiqi Zhang, Xiang Li, Bhiksha Raj, and Huaxiu Yao. 2024b. Autoprm: Automating procedural supervision for multi-step reasoning via controllable question decomposition. arXiv preprint arXiv:2402.11452. find
  12. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  13. Valerio Cosentino, Javier Luis, and Jordi Cabot. 2016. Findings from github: methods, datasets and limitations. In Proceedings of the 13th international conference on mining software repositories, pages 137–141. find
  14. Ian A Crawford. 2015. Lunar resources: A review. Progress in Physical Geography, 39(2):137–167. find
  15. Alexander Cushen, Ariana Bueno, Samuel Carrico, Corrydon Wettstein, Jaykumar Ishvarbhai Adalja, Mengxiang Shi, Naila Garcia, Yuliana Garcia, Mirko Gamba, and Christopher Ruf. 2025. Arc-light: Algorithm for robust characterization of lunar surface imaging for ground hazards and trajectory. Aerospace,… find
  16. Liang Ding, Ruyi Zhou, Ye Yuan, Huaiguang Yang, Jian Li, Tianyi Yu, C Liu, Jian Wang, Shu Li, Haibo Gao, and 1 others. 2022. A 2-year locomotive exploration and scientific investigation of the lunar farside by the yutu-2 rover. Science Robotics, 7(62):eabj6660. find
  17. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, and 1 others. 2024. A survey on in-context learning. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 1107–1128.
  18. N Abu El Samid, Jekanthan Thangavelautham, and G D’Eleuterio. 2008. Infrastructure robotics: A technology enabler for lunar in-situ resource utilization, habitat construction and maintenance. In Proceedings of International Astronautic Conference, pages 2045–2058. find
  19. Alex Ellery. 2020. Sustainable in-situ resource utilization on the moon. Planetary and Space Science, 184:104870. find
  20. European Space Agency. 2023. A2I roadmap for ESA’s missions operations. https://esoc.esa.int/ a2i-roadmap-esas-missions-operations. Accessed: 2025-05-06. ↗
  21. Jessica Flahaut, Carolyn H van der Bogert, Ian A Crawford, and Sebastien Vincent-Bonnieu. 2023. Scientific perspectives on lunar exploration in europe. npj Microgravity, 9(1):50. find
  22. Jeremy D Frank. 2020. Artificial intelligence: Powering human exploration of the moon and mars. In ASCEND 2020, page 4164. find
  23. Gianluca Furano, Antonis Tavoularis, and Marco Rovatti. 2020. Ai in space: Applications examples and challenges. In 2020 IEEE International Symposium on Defect and Fault Tolerance in VLSI and Nanotechnology Systems (DFT), pages 1–6. IEEE. find
  24. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, and 1 others. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. find
  25. Mingqi Gao, Xinyu Hu, Xunjian Yin, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2025. Llm-based nlg evaluation: Current status and challenges. Computational Linguistics, pages 1–27.
  26. Alejandro Gonzalez, Michelle L Peters, Amy Orange, and Bettye Grigsby. 2017. The influence of high-stakes testing on teacher self-efficacy and job-related stress. Cambridge Journal of Education, 47(4):513– 531. find
  27. Clémentine Grethen, Simone Gasparini, Géraldine Morin, Jeremy Lebreton, Lucas Marti, and Manuel Sanchez-Gestido. 2025. Adapting stereo vision from objects to 3d lunar surface reconstruction with the stereolunar dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, page… find
  28. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. find
  29. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  30. Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. 2020. Wiki-40b: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2440–2452. find
  31. Mohammad Amin Habibi, Fateme Aghaei, Zohreh Ta-jabadi, Mohammad Sina Mirjani, Poriya Minaee, and SeyedMohammad Eazi. 2024. The performance of machine learning for prediction of h3k27 m mutation in midline gliomas: a systematic review and meta-analysis. World Neurosurgery, 186:e7–e19. find
  32. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  33. Miro Hodak, David Ellison, Chris Van Buren, Xiaotong Jiang, and Ajay Dholakia. 2023. Benchmarking large language models: opportunities and challenges. In Technology Conference on Performance Evaluation and Benchmarking, pages 77–89. Springer. find
  34. Jie Huang and Kevin Chen-Chuan Chang. 2022. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403.
  35. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, and 1 others. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36:62991–63010…
  36. Babar Hussain, Jiandong Guo, Sidra Fareed, and Subhan Uddin. 2025. Robotics for space exploration: From mars rovers to lunar missions. International Journal of Ethical AI Application, 1(1):1–10. find
  37. Dario Izzo, Gabriele Meoni, Pablo Gómez, Dominik Dold, and Alexander Zoechbauer. 2023. Selected trends in artificial intelligence for space applications. In Artificial Intelligence for Space: AI4SPACE, pages 21–52. CRC Press. find
  38. Steeven Janny, Hervé Poirier, Leonid Antsfeld, Guillaume Bono, Gianluca Monaci, Boris Chidlovskii, Francesco Giuliari, Alessio Del Bue, and Christian Wolf. 2025. Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach. In Proceedings of the Computer Vision and Patte… find
  39. Anthi Koskina, Olga Sykioti, and Manolis Plionis. 2023. Ai-driven innovation and discoveries in space exploration: The need for an adapted regulatory regime. In International Conference on Frontiers of Artificial Intelligence, Ethics, and Multidisciplinary Applications, pages 377–390. Springer. find
  40. Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. 2025. Llm post-training: A deep dive into reasoning large language models. arXiv preprint arXiv:2502.21321. — [arXiv:2502.21321](https://arxiv.org/abs/2502.21321) find
  41. Hanna Kurniawati. 2022. Partially observable markov decision processes and robotics. Annual Review of Control, Robotics, and Autonomous Systems, 5(1):253–277. find
  42. Chunlai Li, Chi Wang, Yong Wei, and Yangting Lin. 2019. China’s present and future lunar exploration program. Science, 365(6450):238–239. find
  43. Yucheng Li, Frank Guerin, and Chenghua Lin. 2023. An open source data contamination report for large language models. arXiv preprint arXiv:2310.17589. find
  44. Yuetai Li, Zhaoyang Sun, Rui Chen, Hao Qian, and Yifan Zhao. 2025a. Small models struggle to learn from strong reasoners. arXiv preprint arXiv:2502.12143.
  45. Zhongyan Li, Shangfu Li, Mengqi Luo, Jhih-Hua Jhong, Wenshuo Li, Lantian Yao, Yuxuan Pang, Zhuo Wang, Rulan Wang, Renfei Ma, and 1 others. 2022. dbptm in 2022: an updated database for exploring regulatory networks and functional associations of protein post-translational modifications. Nucleic acids… find
  46. Zongxia Li, Xiyang Wu, Hongyang Du, Huy Nghiem, and Guangyao Shi. 2025b. Benchmark evaluations, applications, and challenges of large vision language models: A survey. arXiv preprint arXiv:2501.02189, 1. — [arXiv:2501.02189](https://arxiv.org/abs/2501.02189) find
  47. Yangting Lin, Wei Yang, Hui Zhang, Hejiu Hui, Sen Hu, Long Xiao, Jianzhong Liu, Zhiyong Xiao, Zongyu Yue, Jinhai Zhang, and 1 others. 2024. Return to the moon: New perspectives on lunar exploration. Science Bulletin, 69(13):2136–2148. find
  48. Jiayi Liu, Qianyu Zhang, Xue Wan, Shengyang Zhang, Yaolin Tian, Haodong Han, Yutao Zhao, Baichuan Liu, Zeyuan Zhao, and Xubo Luo. 2024. Lusnar: A lunar segmentation, navigation and reconstruction dataset based on muti-sensor for autonomous exploration. arXiv preprint arXiv:2407.06512. — [arXiv:2407.06512](https://arxiv.org/abs/2407.06512) find
  49. Francis Xian Logah, Younho Seong, Jennifer Baanye, Manuella Wilson, Azeez Adamolekun, Sun Yi, and Kelvin Kwakye. 2025. A review of the impact of cognitive workload on reaction time and performance. find
  50. Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15126.
  51. Martin J Losekamm, Janos Biswas, Thibaud Chupin, Michael Deiml, Matthieu Deremetz, Anthony M Evagora, Guillaume Fau, Jessica Flahaut, Jeremi Gancet, Markus Glier, and 1 others. 2022. Assessing the distribution of water ice and other volatiles at the lunar south pole with luvmi-x: a mission concept.… find
  52. Yubo Ma, Zhibin Gou, Junheng Hao, Ruochen Xu, Shuohang Wang, Liangming Pan, Yujiu Yang, Yixin Cao, Aixin Sun, Hany Awadalla, and 1 others. 2024. Sciagent: Tool-augmented language models for scientific reasoning. arXiv preprint arXiv:2402.11451.
  53. David Maranto. 2024. Llmsat: A large language model-based goal-oriented agent for autonomous space exploration. arXiv preprint arXiv:2405.01392. — [arXiv:2405.01392](https://arxiv.org/abs/2405.01392) find
  54. Peter McKenna. 2019. Multiple choice questions: answering correctly and knowing the answer. Interactive Technology and Smart Education, 16(1):59–73. find
  55. IG Mitrofanov, A Bartels, YI Bobrovnitsky, W Boynton, G Chin, H Enos, L Evans, S Floyd, J Garvin, DV Golovin, and 1 others. 2010. Lunar exploration neutron detector for the nasa lunar reconnaissance orbiter. Space science reviews, 150(1):183–207. find
  56. Niklas Muenchhoff, Thomas Bauer, Jihwan Lee, Minseok Kim, and Maximilian Schmid. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393.
  57. NASA. 2024. Artificial intelligence at NASA. https://www.nasa.gov/artificial-intelligence/. Accessed: 2025-05-06. ↗
  58. Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255.
  59. Jay M Patel. 2020. Introduction to common crawl datasets. In Getting structured data from the internet: running web crawlers/scrapers on a big data production scale, pages 277–324. Springer. find
  60. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  61. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. find
  62. Ranjan Sapkota, Shaina Raza, and Manoj Karkee. 2025. Comprehensive analysis of transparency and accessibility of chatgpt, deepseek, and other sota large language models. arXiv preprint arXiv:2502.18505. — [arXiv:2502.18505](https://arxiv.org/abs/2502.18505) find
  63. Martin J Schuster, Marcus G Müller, Sebastian G Brunner, Hannah Lehner, Peter Lehner, Ryo Sakagami, Andreas Dömel, Lukas Meyer, Bernhard Vodermayer, Riccardo Giubilato, and 1 others. 2020. The arches space-analogue demonstration mission: Towards heterogeneous teams of autonomous robots for collabora… find
  64. Rui Pei, Michael Pittman, Pablo A Goloboff, T Alexander Dececchi, Michael B Habib, Thomas G Kaye, Hans CE Larsson, Mark A Norell, Stephen L Brusatte, and Xing Xu. 2020. Potential for powered flight neared by most close avialan relatives, but few crossed its thresholds. Current Biology, 30(20):4033–… find
  65. Brent Sherwood. 2019. Principles for a practical moon base. Acta Astronautica, 160:116–124. find
  66. Michael Pekala, Gregory Canal, Samuel Barham, Milena B Graziano, Morgan Trexler, Leslie Hamilton, Elizabeth Reilly, and Christopher D Stiles. 2025. Towards large language models for lunar mission planning and in situ resource utilization. arXiv preprint arXiv:2504.20125. — [arXiv:2504.20125](https://arxiv.org/abs/2504.20125) find
  67. Xinshuai Song, Weixing Chen, Yang Liu, Weikai Chen, Guanbin Li, and Liang Lin. 2025. Towards long-horizon vision-language navigation: Platform, benchmark and method. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12078–12088. find
  68. Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. 2024. Reasoning with large language models, a survey. arXiv preprint arXiv:2407.11511.
  69. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, and 1 others. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.…
  70. Mirali Purohit, Bimal Gajera, Vatsal Malaviya, Irish Mehta, Kunal Kasodekar, Jacob Adler, Steven Lu, Umaa Rebbapragada, and Hannah Kerner. 2025. Mars-bench: A benchmark for evaluating foundation models for mars science tasks. arXiv preprint arXiv:2510.24010. — [arXiv:2510.24010](https://arxiv.org/abs/2510.24010) find
  71. Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2024. Towards building multilingual language model for medicine. Nature Communications, 15(1):8384.
  72. Rahul Ramachandran, Manil Maskey, Kaylin Bugbee, Mike Little, Elizabeth Fancher, Muthukumaran Ramasubramanian, Bishwaranjan Bhattaacharjee, Raghu Ganti, Avi Sil, Lauren Sanders, and 1 others. 2023. Harnessing large language models for scientific endeavors. In 23rd Meeting of the American Geophysical… find
  73. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. 2024. Ai models collapse when trained on recursively generated data. Nature, 631(8022):755–759.
  74. Richard S Sutton, Andrew G Barto, and 1 others. 1998. Reinforcement learning: An introduction, volume 1. MIT press Cambridge. find
  75. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and 1 others. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.
  76. R Michael Swan, Deegan Atha, Henry A Leopold, Matthew Gildner, Stephanie Oij, Cindy Chiu, and Masahiro Ono. 2021. Ai4mars: A dataset for terrain-aware autonomous driving on mars. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1982–1991. find
  77. John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science, 12(2):257–285. find
  78. Yixin Ye, Zeyu Li, Haoran Zhang, Ming Xu, and Jian Wang. 2025. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387.
  79. John Sweller. 1994. Cognitive load theory, learning difficulty, and instructional design. Learning and instruction, 4(4):295–312. find
  80. Hang Zhang, Jiawei Sun, Renqi Chen, Wei Liu, Zhonghang Yuan, Xinzhe Zheng, Zhefan Wang, Zhiyuan Yang, Hang Yan, Hansen Zhong, and 1 others. 2024. Empowering and assessing the utility of large language models in crop science. Advances in Neural Information Processing Systems, 37:52670–52722. find
  81. Qwen Team. 2024. Qwen2.5-72b-instruct. https://huggingface.co/Qwen/Qwen2.5-72B-Instruct. Accessed: 2025-05-13. ↗
  82. Indhu Varatharajan, Daniel Angerhausen, Eleni Anto-niadou, Valentin Bickel, Mario D’Amore, Michele Faragalli, Ignacio López-Francos, Abhisek Maiti, Ross WK Potter, Carl Shneider, and 1 others. 2021. Artificial intelligence for the advancement of lunar and planetary science and exploration. Bulletin… find
  83. Marco Veneranda, Guillermo Lopez-Reyes, Jose Antonio Manrique-Martinez, Aurelio Sanz-Arranz, Emmanuel Lalla, Menelaos Konstantinidis, Andoni Moral, Jesús Medina, and Fernando Rull. 2020. Ex-omars raman laser spectrometer (rls): Development of chemometric tools to classify ultramafic igneous rocks on… find
  84. Yuwei Wan, Aswathy Ajith, Yixuan Liu, Ke Lu, Clara Grazian, Bram Hoex, Wenjie Zhang, Chunyu Kit, Tong Xie, and Ian Foster. 2024. Sciqag: A framework for auto-generated scientific question answering dataset with fine-grained evaluation. arXiv e-prints, pages arXiv–2405. find
  85. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  86. Xin-Yu Xiao, Yalei Liu, Xiangyu Liu, Zengrui Li, Erwei Yin, and Qianchen Xia. 2025. Lunar twins: We choose to go to the moon with large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 1325–1339. find
  87. Fengna Xu and Jun Ou. 2023. Promoting international cooperation on the international lunar research station: Inspiration from the iter. Acta Astronautica, 203:341–350. find
  88. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  89. Peng Zhang, Wei Dai, Ran Niu, Guang Zhang, Guanghui Liu, Xin Liu, Zheng Bo, Zhi Wang, Haibo Zheng, Chengbao Liu, and 1 others. 2023a. Overview of the lunar in situ resource utilization techniques for future lunar missions. Space: Science & Technology, 3:0037. find
  90. Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. 2023b. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305.12474.
  91. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023c. Safetybench: Evaluating the safety of large language models with multiple choice questions. CoRR.
  92. Xiang Zhao and You Song. 2024. Exploration and application of ai in space science. In ICML 2024 AI for Science Workshop. find
  93. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364.
  94. Gengze Zhou, Yicong Hong, and Qi Wu. 2024. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641–7649. find
  95. Yue Maggie Zhou. 2013. Designing for complexity: Using divisions and hierarchy to manage complex tasks. Organization Science, 24(2):339–355. find
  96. Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. 2023. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36:8958–8974. find
  97. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822. *Figure 8: Operational framework of LLMs in lunar explo… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2026-lunar-bench-evaluating-task-oriented/