Elamur Research⌕
☰
home / papers / 2026 / Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents

Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents

Paper · 2026 · Asaf Yehudai, Lilach Eden and Michal Shmueli-Scheuer

Summary

LLM agents often fail in subtle ways, such as loops, misbehaving sub-agents, or errors that cascade across steps. These problems are hard to spot from final outputs alone. Observability tools like LangSmith and LangFuse record traces but provide only shallow evaluation, and published error taxonomies are fixed and cannot adapt to new tasks. The authors present Agentic CLEAR, an open-source framework that produces textual diagnostics at three levels: system, node, and trace.

The pipeline has two stages. First, an LLM judge scores and critiques each LLM call, critiques each trace as a whole, and checks the trace against rubrics it generates for the task. Second, the earlier CLEAR method clusters these critiques into recurring issues for each node and for the whole system. Results are explored through an interactive dashboard.

Across four benchmarks and seven agent configurations, the discovered issues correspond reasonably well to human-annotated TRAIL error categories, and the judge scores help predict whether a task succeeded. This gives developers a way to diagnose agents without writing tests or rubrics by hand.

Key points

  • Distributed as a PyPI package (pip install clear-eval, Apache 2.0). It reads OpenTelemetry/LangFuse-formatted traces, is configured through a YAML file, and runs with a single CLI command.
  • Evaluation covered AppWorld (CUGA agent), GAIA, SWE-bench Verified Mini, and τ²-Bench, using GPT-5 and OSS-120B as judges.
  • TRAIL alignment used 12 of 20 categories (the non-execution ones, which account for 94% of labels) across 117 traces:
    • GPT-5 reached macro-F1 0.459 and micro-F1 0.497.
    • A frequency-weighted random baseline reached micro-F1 0.459 but only 0.199 macro-F1.
    • GPT-5's 15 issues covered all 12 categories; OSS-120B's 12 issues covered 10.
  • For predicting success (AUC), the trace-level score was generally the strongest of three methods (trace, rubric, step-wise).
    • On AppWorld every method exceeded 0.75, and GPT-5 reached at least 0.82.
    • On τ²-Bench no method went above 0.62.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States154 works · 72,606 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksDecision Sciences · 249 worksComputer Networks and Communications · 100 worksInformation Systems and Management · 52 works

Cite

Cite as Asaf Yehudai, Lilach Eden and Michal Shmueli-Scheuer. 2026. Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-demo.74

@inproceedings{yehudai2026agentic,
  title = {{Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents}},
  author = {Asaf Yehudai, Lilach Eden and Michal Shmueli-Scheuer},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-demo.74},
  url = {https://doi.org/10.18653/v1/2026.acl-demo.74}
}

References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Liu, Botao Yu, Amit Arora, Dongyoon Hahm, Harsh Trivedi, Huan Sun, and 12 others. 2026. Holistic agent leaderboard: The missing infrastructure for AI agent evaluation. In The Fourteenth International Conference on Learning Representations. find
  2. Anthropic. 2025. Claude-code. find
  3. Elron Bandel, Asaf Yehudai, Lilach Eden, Yehoshua Sagron, Yotam Perlitz, Elad Venezian, Natalia Razinkov, Natan Ergas, Shlomit Shachor Ifergan, Segev Shlomov, Michal Jacovi, Leshem Choshen, Liat Ein-Dor, Yoav Katz, and Michal Shmueli-Scheuer. 2026a. General agent evaluation. Preprint, arXiv:2602.229… — [arXiv:2602.22953](https://arxiv.org/abs/2602.22953) find
  4. Elron Bandel, Asaf Yehudai, Alexandre Lacoste, Avijit Ghosh, Graham Neubig, Margaret Mitchell, Michal Shmueli-Scheuer, and Leshem Choshen. 2026b. Agentic systems should be general. SSRN Electronic Journal. find
  5. Elron Bandel, Asaf Yehudai, and Michal Shmueli-Scheuer. 2026c. Ready for general agents? let’s test it. In ICLR Blogposts 2026. Https://iclr-blogposts.github.io/2026/blog/2026/general-agent-evaluation/. find
  6. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. τ 2 -bench: Evaluating conversational agents in a dual-control environment. Preprint, arXiv:2506.07982.
  7. Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2026. Why do multi-agent LLM systems fail? In The Thirty-ninth Annual Conference on Neural Inf…
  8. Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian. 2025. Trail: Trace reasoning and agentic issue localization. arXiv preprint arXiv:2505.08638. find
  9. Ariel Gera, Odellia Boni, Yotam Perlitz, Roy Bar-Haim, Lilach Eden, and Asaf Yehudai. 2025. Justrank: Benchmarking llm judges for system ranking. Preprint, arXiv:2412.09569.
  10. Alireza Ghafarollahi and Markus J Buehler. 2025. Sciagents: automating scientific discovery through bioinspired multi-agent intelligent graph reasoning. Advanced Materials, 37(22):2413523. find
  11. Harbor Framework Team. 2026. Harbor: A framework for evaluating and optimizing agents and models in container environments. find
  12. Alexandre Lacoste, Nicolas Gontier, Oleh Shliazhko, Aman Jaiswal, Kusha Sareen, Shailesh Nanisetty, Joan Cabezas, Manuel Del Verme, Omar G. Younis, Simone Baratta, Matteo Avalle, Imene Kerboua, Xing Han Lù, Elron Bandel, Michal Shmueli-Scheuer, Asaf Yehudai, Leshem Choshen, Jonathan Lebensold, Sean… — [arXiv:2603.15798](https://arxiv.org/abs/2603.15798) find
  13. LangFuse. 2023. Langfuse: Observability for ai applications. find
  14. LangSmith. 2023. Langsmith: Evaluation framework for ai applications. find
  15. Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stanczak, Peter Shaw, Christopher Pal, and Siva Reddy. 2025. Agentrewardbench: Evaluating automatic evaluations of web agent trajectories. In Second Conference on Language Modeling. find
  16. Sami Marreed, Alon Oved, Avi Yaeli, Segev Shlomov, Ido Levy, Offer Akrabi, Aviad Sela, Asaf Adi, and Nir Mashkif. 2025. Towards enterprise-ready computer using generalist agent. Preprint, arXiv:2503.01861. — [arXiv:2503.01861](https://arxiv.org/abs/2503.01861) find
  17. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. Preprint, arXiv:2311.12983.
  18. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and more. 2025. gpt-oss-120b & gpt-oss-20b model card.…
  19. OpenAI. 2025. Chatgpt agent: bridging research and action. find
  20. Aymeric Roucher, Albert Villanova del Moral, Merve Noyan, Thomas Wolf, and Clémentine Fourrier. 2025. Open-source DeepResearch – Freeing our search agents. Accessed: 2025-02-04. find
  21. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? Preprint, arXiv:2310.06770.
  22. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539–68551.
  23. Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj Oruganty, Sophie Luskin, Kangheng Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aide… — OpenAI GPT-5 System Card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yehudai-2026-agentic-clear-automating-multi-level/