Elamur Research⌕
☰
home / papers / 2026 / SGVEF-LOOP: Coverage-Guided Progressive Topological Exploration and Fact-Grounded Metamorphic Evaluation for MCP Agents

SGVEF-LOOP: Coverage-Guided Progressive Topological Exploration and Fact-Grounded Metamorphic Evaluation for MCP Agents

Paper · 2026 · Zenghao Liu and Yansong Zhang

Summary

The Model Context Protocol (MCP) now links agents to thousands of servers. This creates a tool space too large and combinatorial for hand-built or template-based benchmarks. The authors present SGVEF-LOOP, which treats valid tool sequences as an unknown layered directed graph, the Latent Tool Transition Graph. An edge in this graph is valid only if the next tool's arguments can be derived from the task context or earlier outputs, and the step makes sense in a real workflow.

A feedback-driven sampler steers task generation toward tools and transitions that have been explored least. For each task, the system builds a metamorphic pair: two differently worded queries with the same intent and the same expected tool trace. This lets robustness be checked without a ground-truth oracle. An LLM judge plus human review filters the pairs, and accepted pairs go into a vector database as seeds for later rounds. Completeness is measured against a Chao1 estimate of the reachable transition space.

The resulting benchmark covers every tool and most estimated transitions. Testing eight models reveals clear capability tiers and fragility under rewording, offering a scalable recipe for evaluating agents in changing tool ecosystems.

Key points

  • Coverage: the framework reaches 100% tool (node) coverage by iteration 15 and 80.54% of the Chao1-estimated transition bound by iteration 60. In total it mines 1,080 effective tool transition paths.
  • Diminishing returns: coverage gains drop from about 37.7% in rounds 1–15 to about 5.4% in rounds 45–60. The authors think the unexplored 19.46% consists mostly of rare, deep multi-hop transitions.
  • Consistency results:
    • Qwen3-Max is the most consistent model (MTC 0.602, Function Agreement 0.688).
    • DeepSeek R1 has a very low Sequence Agreement (0.107). It also has the lowest TFD (0.163) with a TCS of 0.209, which suggests it gives up early rather than recovering.
  • Answer–execution gap:
    • Claude Haiku 4.5 scores highest on AR (0.853) and UIC (0.683) but only 0.263 on TCS. Its |ECG| of 0.361 is the largest, which hints that its answers rely on parametric knowledge rather than tool results.
    • Claude 3.5 Sonnet and Kimi K2 keep gaps near 0.20. Kimi K2 also shows a high TFD (0.455).
  • Failure patterns and availability: metamorphic testing exposes three failure patterns: incomplete planning, sensitivity to irrelevant context, and unstable tool selection. Code and data are available at github.com/zh6zh/SGVEF-LOOP.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Renmin University of ChinaBeijing · China114 works · 16,118 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSoftware · 92 worksComputer Networks and Communications · 64 works

Cite

Cite as Zenghao Liu and Yansong Zhang. 2026. SGVEF-LOOP: Coverage-Guided Progressive Topological Exploration and Fact-Grounded Metamorphic Evaluation for MCP Agents. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{liu2026sgvef,
  title = {{SGVEF-LOOP: Coverage-Guided Progressive Topological Exploration and Fact-Grounded Metamorphic Evaluation for MCP Agents}},
  author = {Zenghao Liu and Yansong Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1224},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1224}
}

References

49 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024a. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet. ↗
  2. Anthropic. 2024b. Introducing the model context protocol. https://www.anthropic.com/news/model -context-protocol. ↗
  3. Anthropic. 2025a. Claude haiku 4.5. https://www.anthropic.com/claude/haiku. ↗
  4. Anthropic. 2025b. Claude sonnet 4. https://www.an thropic.com/news/claude-4. ↗
  5. Anthropic. 2025c. Donating the model context protocol and establishing the agentic ai foundation. https://www.anthropic.com/news/donating-the-m odel-context-protocol-and-establishing-o f-the-agentic-ai-foundation. ↗
  6. Anne Chao. 1984. Nonparametric estimation of the number of classes in a population. Scandinavian Journal of Statistics, 11:265–270. find
  7. Francisco Eiras, Eliott Zemour, Eric Lin, and Vaikkunth Mugunthan. 2025. Know thy judge: On the robustness meta-evaluation of llm safety judges. arXiv preprint arXiv:2503.04474. — [arXiv:2503.04474](https://arxiv.org/abs/2503.04474) find
  8. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. Ragas: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158.
  9. Xuanqi Gao, Siyi Xie, Juan Zhai, Shiqing Ma, and Chao Shen. 2025. Mcp-radar: A multidimensional benchmark for evaluating tool use capabilities in large language models. arXiv preprint arXiv:2505.16700. — [arXiv:2505.16700](https://arxiv.org/abs/2505.16700) find
  10. Thai Hoang, Kung-Hsiang Huang, Shirley Kokane, Jianguo Zhang, Zuxin Liu, Ming Zhu, Jake Grigsby, Tian Lan, Michael S. Ryoo, Chien-Sheng Wu, Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. 2025. Lam simulator: Advancing data generation for large action model train… find
  11. Yuheng Huang, Da Song, Zhenlan Ji, Shuai Wang, and Lei Ma. 2025. Evaluating llms on sequential api call through automated test generation. arXiv preprint arXiv:2507.09481. — [arXiv:2507.09481](https://arxiv.org/abs/2507.09481) find
  12. Kimi. 2025. Kimi-k2. https://github.com/kimi-k 2/Kimi-K2. ↗
  13. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244.
  14. Anne Chao. 1987. Estimating the population size for capture-recapture data with unequal catchability. Biometrics, pages 783–791. find
  15. Ningke Li, Yuekang Li, Yi Liu, Ling Shi, Kailong Wang, and Haoyu Wang. 2024. Drowzee: Metamorphic testing for fact-conflicting hallucination detection in large language models. Proceedings of the ACM on Programming Languages, 8(OOPSLA2):1843–1872. find
  16. Tsong Yueh Chen, Shing Chi Cheung, and Siu ming Yiu. 2020. Metamorphic testing: A new approach for generating next test cases. ArXiv, abs/2002.12543. — [arXiv:2002.12543](https://arxiv.org/abs/2002.12543) find
  17. Hongliang Liang, Xiaoxiao Pei, Xiaodong Jia, Wuwei Shen, and Jian Zhang. 2018. Fuzzing: State of the art. IEEE Transactions on Reliability, 67(3):1199– 1218. find
  18. Steven Cho, Stefano Ruberto, and Valerio Terragni. 2025. Metamorphic testing of large language models for natural language processing. arXiv preprint arXiv:2511.02108. — [arXiv:2511.02108](https://arxiv.org/abs/2511.02108) find
  19. DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  20. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Yuxian Gu, Han Ding, Kai Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Shengqi Shen, Tianjun Zhang, Sheng Shen, and 5 others. 2023. Agentbench: Evaluating llms as agents. arXiv preprint ar…
  21. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  22. Zhiwei Liu, Jielin Qiu, Shiyu Wang, Jianguo Zhang, Zuxin Liu, Roshan Ram, Haolin Chen, Weiran Yao, Huan Wang, Shelby Heinecke, Silvio Savarese, and Caiming Xiong. 2025. Mcpeval: Automatic mcp-based deep evaluation for ai agent models. In Proceedings of the 2025 Conference on Empirical Methods in Nat… find
  23. Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. 2025. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. In Findings of the Association for Com…
  24. Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. 2025. Mcp-universe: Benchmarking large language models with real-world model context protocol servers. arXiv preprint arXiv:2508.14704. — [arXiv:2508.14704](https://arxiv.org/abs/2508.14704) find
  25. Barton P Miller, Lars Fredriksen, and Bryan So. 1990. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32–44. find
  26. Robert Osazuwa Ness, Katie Matton, Hayden S. Helm, Sheng Zhang, Junaid Bajwa, Carey E. Priebe, and Eric Horvitz. 2024. Medfuzz: Exploring the robustness of large language models in medical question answering. ArXiv, abs/2406.06573. — [arXiv:2406.06573](https://arxiv.org/abs/2406.06573) find
  27. OpenAI. 2024a. Gpt-4o mini: Advancing cost-effective artificial intelligence. https://openai.com/zh-Hant-HK/index/gpt-4o-mini-advanci ng-cost-efficient-intelligence/. ↗
  28. OpenAI. 2024b. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/. ↗
  29. OpenAI. 2025. Gpt-5 debuts. https://openai.com/zh-Hant-HK/gpt-5/. ↗
  30. Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning. find
  31. Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. Toolllm: Facilitating large language models to master 16000+ real-world api…
  32. Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. Ares: An automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… find
  33. Guangyu Wang, Jianhong Liu, Meilin Zhou, Xiaoming Chen, Lihua Zhang, and Zhihao Sun. Toolbench 2.0: Evaluating long-horizon and multi-step tool use in llms. find
  34. Guanyu Wang, Yuekang Li, Yi Liu, Gelei Deng, Tianlin Li, Guosheng Xu, Yang Liu, Haoyu Wang, and Kailong Wang. 2024a. Metmap: Metamorphic testing for detecting false vector matching problems in llm augmented generation. 2024 IEEE/ACM First International Conference on AI Foundation Models and Software… find
  35. Wenxuan Wang, Juluan Shi, Zhaopeng Tu, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael R Lyu. 2024b. The earth is flat? unveiling factual errors in large language models. arXiv preprint arXiv:2401.00761. — [arXiv:2401.00761](https://arxiv.org/abs/2401.00761) find
  36. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long p… find
  37. Zhenting Wang, Qi Chang, Hemani Patel, Sanjana Biju, Chen Wu, Quan Liu, Aolin Ding, Alireza Rezazadeh, Ankit Shah, Yujia Bao, and Eugene Siow. 2025. Mcp-bench: Benchmarking tool-using llm agents with complex real-world tasks via mcp servers. arXiv preprint arXiv:2508.20453. — [arXiv:2508.20453](https://arxiv.org/abs/2508.20453) find
  38. Zijian Wu, Xiangyan Liu, Xinyuan Zhang, Lingjun Chen, Fanqing Meng, Lingxiao Du, Yiran Zhao, Fan Zhang, Yaoqi Ye, Jiawei Wang, Zirui Wang, Jinjie Ni, Yufan Yang, Arvin Xu, and Michael Shieh. 2025. Mcpmark: A benchmark for stress-testing realistic and comprehensive mcp use. arXiv preprint arXiv:2509.… — [arXiv:2509.24002](https://arxiv.org/abs/2509.24002) find
  39. Qwen. 2025. Qwen3-max. https://qwen.ai/blog ?id=qwen3-max. ↗
  40. Yun-Yu Yan, Shihe Wang, Jiajun Du, Yexuan Yang, Yuxuan Shan, Qichen Qiu, Xianqing Jia, Xing Wang, Xin Yuan, Xu Han, Maosheng Qin, Yinxiao Chen, Chen Peng, Shangguang Wang, and Mengwei Xu. 2025. Mcpworld: A unified benchmarking testbed for api, gui, and hybrid computer use agents. arXiv preprint arXi… find
  41. Mark Russinovich, Ahmed Salem, and Ronen Eldan. 2024. Great, now write an article about that: The crescendo multi-turn llm jailbreak attack. ArXiv, abs/2404.01833. find
  42. Yixuan Yang, Daoyuan Wu, and Yufan Chen. 2025. Mcpsecbench: A systematic security benchmark and playground for testing model context protocols. arXiv preprint arXiv:2508.13220. — [arXiv:2508.13220](https://arxiv.org/abs/2508.13220) find
  43. Dongyu Yao, Jianshu Zhang, Ian G. Harris, and Marcel Carlsson. 2023. Fuzzllm: A novel and universal fuzzing framework for proactively discovering jailbreak vulnerabilities in large language models. ArXiv, abs/2309.05274. — [arXiv:2309.05274](https://arxiv.org/abs/2309.05274) find
  44. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. ArXiv, abs/2406.12045.
  45. Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2023. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253.
  46. Jiahao Yu, Yangguang Shao, Hanwen Miao, Junzheng Shi, and Xinyu Xing. 2024. Promptfuzz: Harnessing fuzzing techniques for robust testing of prompt injection in llms. ArXiv, abs/2409.14729. — [arXiv:2409.14729](https://arxiv.org/abs/2409.14729) find
  47. Yukai Zhao, Menghan Wu, Xing Hu, and Xin Xia. 2025. Hfuzzer: Testing large language models for package hallucinations via phrase-based fuzzing. ArXiv, abs/2509.23835. — [arXiv:2509.23835](https://arxiv.org/abs/2509.23835) find
  48. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:…
  49. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-sgvef-loop-coverage-guided-progressive/