Elamur Research⌕
☰
home / papers / 2026 / AutoTaskEval: Towards Domain-Specific and Fine-Grained Evaluation for LLMs

AutoTaskEval: Towards Domain-Specific and Fine-Grained Evaluation for LLMs

Paper · 2026 · Qingqing Lyu, Linjuan Wu, Yongliang Shen et al.

Summary

Static, manually built benchmarks are expensive, cover few tasks, and lose discriminative power as LLMs improve. Earlier automatic generators also only fill in task types defined in advance. AutoTaskEval instead builds domain-specific benchmarks from raw, unstructured corpora, using only open-source models. The pipeline has three stages:

  1. Task discovery. It samples diverse passages with K-Center-Greedy and derives scenarios from them. Each scenario is paired with two meta-capabilities from an extended Bloom's Taxonomy to define a task.
  2. Context enrichment. Teacher and student agents question each other in a Socratic loop to surface key concepts. These concepts are then used to retrieve supporting text from the corpus.
  3. Instance synthesis. The teacher writes questions, the student answers, and a reviewer judges quality. The loop repeats, pushing each new item to be harder than the last.

The authors tested the framework mainly on Chinese law, using LawBench as the reference. Model rankings on the generated items closely tracked rankings on the human-curated benchmark. The items were also harder than those from competing automatic generators. A second study on noisy e-commerce reviews shows the approach also works where no domain ontology exists. This matters for any team that needs evaluation sets for specialized fields, since few benchmarks cover them.

Key points

  • Capability taxonomy: two levels, with the six Bloom dimensions (Remember, Understand, Apply, Analyze, Evaluate, Create) split into 27 fine-grained sub-skills.
  • Legal validation: four LawBench tasks were regenerated and nine models were scored on them. Model rankings on the generated items correlated with LawBench rankings at Spearman 0.9.
  • Difficulty: the generated items were harder than those from Zero-shot Benchmarking (ZSB) and BenchAgents, and roughly as hard as LawBench. The "Ours-Hard" subset was harder than LawBench.
  • Capability profiles: larger Qwen2.5 models gained most at the Create level. Qwen2.5-7B-Instruct was strong in Analyze but weak in C1 (Ideation) and C5 (Prototyping & Refinement).
  • AutoCommerceEval:
    • Task diversity, measured as Shannon entropy over capabilities, reached 4.36, higher than the structured legal benchmarks.
    • Models were tested on three tasks with 60 instances each, across the Qwen2.5, gemma-3 and llama-3 families.
    • Within each family, scores generally rose with model size.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Qingqing Lyu, Linjuan Wu, Yongliang Shen, Hengwei Liu, Hao Li, Shengpei Jiang, Yin Zhang and Weiming Lu. 2026. AutoTaskEval: Towards Domain-Specific and Fine-Grained Evaluation for LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.280

@inproceedings{lyu2026autotaskeval,
  title = {{AutoTaskEval: Towards Domain-Specific and Fine-Grained Evaluation for LLMs}},
  author = {Qingqing Lyu, Linjuan Wu, Yongliang Shen, Hengwei Liu, Hao Li, Shengpei Jiang, Yin Zhang and Weiming Lu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.280},
  url = {https://doi.org/10.18653/v1/2026.acl-long.280}
}

References

24 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Claude sonnet 4. https://huggingface.co/CometAPI/Claude_Sonnet4. Accessed: 2025-10-06. ↗
  2. Natasha Butt, Varun Chandrasekaran, Neel Joshi, Besmira Nushi, and Vidhisha Balachandran. 2024. Benchagents: Automated benchmark creation with agent interaction. Preprint, arXiv:2410.22584. — [arXiv:2410.22584](https://arxiv.org/abs/2410.22584) find
  3. Yulong Chen, Yang Liu, Jianhao Yan, Xuefeng Bai, Ming Zhong, Yinghao Yang, Ziyi Yang, Chenguang Zhu, and Yue Zhang. 2024. See what llms cannot answer: A self-challenge framework for uncovering llm weaknesses. Preprint, arXiv:2408.08978. — [arXiv:2408.08978](https://arxiv.org/abs/2408.08978) find
  4. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating llms by human preference. Preprint, arXiv:2403.04132.
  5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  6. Google DeepMind. 2025. Gemini 2.5 pro: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. https://arxiv.org/abs/2507.06261. Accessed: 2025-10-06. ↗
  7. Hu Ding, Haikuo Yu, and Zixiu Wang. 2019. Greedy strategy works for k-center clustering with outliers and coreset construction. arXiv preprint arXiv:1901.08219. — [arXiv:1901.08219](https://arxiv.org/abs/1901.08219) find
  8. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. Lawbench: Benchmarking legal knowledge of large language models. Preprint, arXiv:2309.16289.
  9. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, and 21… — LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
  10. Jingyuan Qi, Zhiyang Xu, Ying Shen, Minqian Liu, Di Jin, Qifan Wang, and Lifu Huang. 2023. The art of SOCRATIC QUESTIONING: Recursive thinking with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4177–4199, Singapore. Associatio… find
  11. Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. 2022. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. Preprint, arXiv:2207.00220.
  12. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  13. Kunze Li and Yu Zhang. 2024. Planning first, question second: An LLM-guided method for controllable question generation. In Findings of the Asso-Roucher. 2023. ciation for Computational Linguistics: ACL 2024, Aymeric amazon p roduct r eviews d ataf initi. . pages 4715–4729, Bangkok, Thailand. Associ… find
  14. Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang, and Tatsunori Hashimoto. 2025. Autobencher: Towards declarative benchmark construction. Preprint, arXiv:2407.08351. find
  15. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  16. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605. find
  17. AI Meta. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta. com/blog/llama-4-multimodal-intelligence/, checked on, 4(7):2025. ↗
  18. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  19. David Owen. 2024. How predictable is language model benchmark performance? Preprint, arXiv:2401.04757. find
  20. Park, Jiatong Yu, Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, and Anirudh Goyal. 2025. Ai-assisted generation of difficult math questions. Preprint, arXiv:2407.21009. find
  21. Qwen Team. 2024. Qwen2.5: A party of foundation models. Fang-Fang Zhao, Han-Jie He, Jia-Jian Liang, Jingyun Cen, Yun Wang, Hongjie Lin, Feifei Chen, Tai-Ping Li, Jian-Feng Yang, Lan Chen, and 1 others. 2025a. Benchmarking the performance of large language models in uveitis: a comparative analysis of… find
  22. Yilun Zhao, Kaiyan Zhang, Tiansheng Hu, Sihong Wu, Ronan Le Bras, Taira Anderson, Jonathan Bragg, Joseph Chee Chang, Jesse Dodge, Matt Latzke, Yixin Liu, Charles McGrady, Xiangru Tang, Zihang Wang, Chen Zhao, Hannaneh Hajishirzi, Doug Downey, and Arman Cohan. 2025b. Sciarena: An open evaluation plat…
  23. Arkil Patel, Siva Reddy, and Dzmitry Bahdanau. 2025. How to get your llm to generate challenging problems for evaluation. Preprint, arXiv:2502.14678. find
  24. José Pombal, Nuno M. Guerreiro, Ricardo Rei, and André F. T. Martins. 2025. Zero-shot benchmarking: A framework for flexible and scalable automatic evaluation of language models. Preprint, arXiv:2504.01001. — [arXiv:2504.01001](https://arxiv.org/abs/2504.01001) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lyu-2026-autotaskeval-domain-specific-fine-grained/