Elamur Research⌕
☰
home / papers / 2026 / AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge

AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge

Paper · 2026 · Karen Zhou and Chenhao Tan

Summary

LLM-as-a-Judge evaluation with checklists of yes/no criteria has grown popular, but existing approaches each use their own code, prompts and scoring rules. This makes them hard to compare or reuse. AutoChecklist is an open-source Python library that puts these approaches into one framework. It sorts generation methods into five abstractions by how they derive criteria. Instance-level generators (direct, contrastive) build one checklist per input. Corpus-level generators (inductive, deductive, interactive) build one shared checklist for a whole dataset. Any generator can be chained with optional refiners and a single configurable scorer, and new setups need only Markdown prompt templates. The library includes ten pipelines that reimplement published methods. Users can work through a CLI, a local web UI or the Python API. Checklists produced by the library agreed with human judgments on RewardBench and SummEval. A case study on ICLR 2019 rebuttals showed the library can move to a new domain without code changes. The toolkit makes it easier to compare checklist methods systematically and to use checklists for alignment, RL rewards or self-correction.

Key points

  • Generator taxonomy and pipelines: five generator types (DIRECT, CONTRASTIVE, INDUCTIVE, DEDUCTIVE, INTERACTIVE) underlie ten built-in pipelines:
    • TICK, RocketEval and RLCF direct
    • RLCF candidate-based variants and CRG-style pairwise/listwise
    • CheckEval, InteractEval and a feedback-based pipeline
  • Refiners and scoring:
    • Refiners include a Deduplicator, Tagger, UnitTester and a beam-search Selector.
    • ChecklistScorer combines three scoring schemes: pass rate, weighted score and logprob-normalized score.
    • It runs in batch or per-item mode and reports macro averages plus DRFR.
  • Backends: the library can call models through OpenAI or OpenRouter, or through vLLM, whose VLLMOfflineClient runs inference on a local GPU. The package installs via pip and is released under Apache 2.0.
  • Rebuttal case study: the authors scored 100 ICLR 2019 review–rebuttal pairs from OpenReview.
    • DEDUCTIVE had the strongest correlation with reviewer ratings (Spearman 0.267, p < .01) and an ROC-AUC of 0.668 for predicting rating changes.
    • The corpus-level methods (DEDUCTIVE and INDUCTIVE) were the only ones that significantly predicted rating changes.
    • DIRECT best separated accepted from rejected papers.
  • Limitations: the authors changed the original prompts to standardize the pipelines. They note that prompts may need iteration across models and that API use costs money. Multi-stage human-in-the-loop corpus pipelines are not yet fully supported.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ChicagoChicago · United States52 works · 50,893 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Karen Zhou and Chenhao Tan. 2026. AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-demo.51

@inproceedings{zhou2026autochecklist,
  title = {{AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge}},
  author = {Karen Zhou and Chenhao Tan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-demo.51},
  url = {https://doi.org/10.18653/v1/2026.acl-demo.51}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Seong Yeub Chu, Jong Woo Kim, and Mun Yong Yi. 2025. Think together and work better: Combining humans’ and llms’ think-aloud outcomes for effective text evaluation. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–23. find
  2. Jonathan Cook, Tim Rocktäschel, Jakob Nicolaus Foerster, Dennis Aumiller, and Alex Wang. 2024. TICKing all the boxes: Generated checklists improve LLM evaluation and generation. In Language Gamification - NeurIPS 2024 Workshop. find
  3. Nachum Dershowitz and Rakesh M. Verma. 2023. Rebutting rebuttals. Commun. ACM, 66(9):35–41. find
  4. Jacob Dineen, Aswin Rrv, Qin Liu, Zhikun Xu, Xiao Ye, Ming Shen, Zhaonan Li, Shijie Lu, Chitta Baral, Muhao Chen, and Ben Zhou. 2025. QA-LIGN: Aligning LLMs through constitutionally decomposed QA. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 20619– 20642, Suzhou, C… find
  5. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. find
  6. Momoka Furuhashi, Kouta Nakayama, Takashi Kodama, and Saku Sugawara. 2025. Are checklists really useful for automatic evaluation of generative tasks? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 10641–10664. find
  7. Yang Gao, Steffen Eger, Ines Hanselowski, and Iryna Gurevych. 2019. Does my rebuttal matter? insights from a major NLP conference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1274–1290. Assoc… find
  8. Zhitao He, Yiran Lyu, and Pascale Fung. 2026. Dancing in chains: Strategic persuasion in academic rebuttal via theory of mind. In The Fourteenth International Conference on Learning Representations. find
  9. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2025. RewardBench: Evaluating reward models for language modeling. In Findings of the Association for Computational… find
  10. Yukyung Lee, JoongHoon Kim, Jaehee Kim, Hyowon Cho, Jaewook Kang, Pilsung Kang, and Najoung Kim. 2025. CheckEval: A reliable LLM-as-a-judge framework for evaluating text generation using checklists. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Associatio…
  11. Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. Preprint, arXiv:2406.04770.
  12. Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. 2026. Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment. Preprint, arXiv:2510.07743.
  13. Qianli Ma, Chang Guo, Zhiheng Tian, Siyu Wang, Jipeng Xiao, Yuanhao Yue, and Zhipeng Zhang. 2026. Paper2rebuttal: A multi-agent framework for transparent author response assistance. Preprint, arXiv:2601.14171.
  14. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. InFoBench: Evaluating instruction following ability in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13025– 1304…
  15. Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, and Kai Chen. 2024. Hellobench: Evaluating long text generation capabilities of large language models. Preprint, arXiv:2409.161… — [arXiv:2409.16191](https://arxiv.org/abs/2409.16191) find
  16. Hyun Ryu, Doohyuk Jang, Hyemin S. Lee, Joonhyun Jeong, Gyeongman Kim, Donghyeon Cho, Gyouk Chu, Minyeong Hwang, Hyeongwon Jang, Changhun Kim, Haechan Kim, Jina Kim, Joowon Kim, Yoonjeon Kim, Kwanhyung Lee, Chanjae Park, Heecheol Yun, Gregor Betz, and Eunho Yang. 2025. Reviewscore: Misinformed peer r… — [arXiv:2509.21679](https://arxiv.org/abs/2509.21679) find
  17. Abdelrahman Sadallah, Tim Baumgärtner, Iryna Gurevych, and Ted Briscoe. 2025. The good, the bad and the constructive: Automatically measuring peer review’s utility for authors. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 28991–29021, Suzhou, China… find
  18. Clemencia Siro, Pourya Aliannejadi, and Mohammad Aliannejadi. 2026. Learning to judge: LLMs designing and applying evaluation rubrics. In Findings of the European Chapter of the Association for Computational Linguistics.
  19. Vijay Viswanathan, Yanchao Sun, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. 2025. find
  20. Checklists are better than reward models for aligning language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
  21. Yuxuan Wan, Tianqing Fang, Zaitang Li, Yintong Huo, Wenxuan Wang, Haitao Mi, Dong Yu, and Michael R. Lyu. 2026. Inference-time scaling of verification: Self-evolving deep research agents via test-time rubric-guided verification. arXiv preprint arXiv:2601.15808.
  22. Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, and Jianghong Ma. 2025. Rocketeval: Efficient automated LLM evaluation via grading checklist. In The Thirteenth International Conference on Learning Representations. find
  23. Zhichao Yang, Sepehr Janghorbani, Dongxu Zhang, Jun Han, Qian Qian, Andrew Ressler II, Gregory D. Lyng, Sanjit Singh Batra, and Robert E. Tillman. 2026. Health-SCORE: Towards scalable rubrics for improving health-LLMs. arXiv preprint arXiv:2601.18706. — [arXiv:2601.18706](https://arxiv.org/abs/2601.18706) find
  24. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, and 1 others. 2024. Justice or prejudice? quantifying biases in llm-as-ajudge. arXiv preprint arXiv:2410.02736.
  25. Karen Zhou, John Michael Giorgi, Pranav Mani, Peng Xu, Davis Liang, and Chenhao Tan. 2025. From feedback to checklists: Grounded evaluation of AI-generated clinical notes. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. Association for Compu… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-autochecklist-composable-pipelines-checklist-generation/