Elamur Research⌕
☰
home / papers / 2026 / Do NOT Classify and Count: Hybrid Attribute Control Success Evaluation

Do NOT Classify and Count: Hybrid Attribute Control Success Evaluation

Paper · 2026 · Felix Matthias Saaro, Pius von Däniken, Mark Cieliebak et al.

Summary

Controlled generation systems are usually judged by running an off-the-shelf classifier or detector on their outputs and reporting the share that matches the requested attribute. The authors treat this as a quantification problem and argue that this "classify and count" (CC) practice is unreliable. Its estimate is distorted by the classifier's tpr and fpr on unseen generated data, so different classifiers give different success rates, and none match human judgement.

As a remedy, they adapt Bayesian Classify and Count (BCC), from earlier work by von Däniken et al., to controlled generation. BCC models the true success rate, tpr and fpr as latent variables. A small human-labelled subset anchors these quantities, while the many classifier-only predictions add likelihood evidence.

The method is tested on text (sentiment-conditioned stories) and text-to-image tasks such as object counts and colour scenes. Across classifiers, BCC gives consistent estimates that agree with human labels. However, each classifier output carries little information compared with a human label. This means careful human annotation stays necessary, and comparisons between systems should account for uncertainty.

Key points

  • New benchmark: a two-modality testbed with 600 human-rated and 60,000 classifier-rated samples. Code, data and annotation templates are released under CC-BY-4.0 (generative-model-evaluation-toolkit on GitHub).
  • Why CC is biased: the CC estimate behaves roughly as tpr·p + fpr·(1−p). For example, with tpr = 0.9, fpr = 0.1 and a true rate of 0.3, CC reports 0.34.
  • "Human-equivalent value" metric: this new measure, based on effective additional samples (EAS), shows that thousands of classifier ratings are worth only tens of human labels. The best text metric (GPT-5) adds at most about 103.55 human-equivalent labels, weaker ones (FIB, DSS) roughly 10–60, and the image detector DTR up to 269.5.
  • Simulations: with a budget of 100 human labels and 10,000 classifier scores, the per-sample value is only about 10⁻³. At B = 500, GPT-5 reaches V = 0.048, equal to about 480 extra human labels.
  • System comparison and limitations: systems are ranked via posterior probabilities P(p(i) > p(j)), whereas CC's overconfidence can yield wrong rankings. Stated limitations include toy datasets, weak public classifiers, and no modelling of per-class difficulty (neutral sentiment was harder for the generators).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓ZHAW Zurich University of Applied SciencesWinterthur · Switzerland6 works · 181 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Felix Matthias Saaro, Pius von Däniken, Mark Cieliebak and Jan Milan Deriu. 2026. Do NOT Classify and Count: Hybrid Attribute Control Success Evaluation. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.48

@inproceedings{saaro2026do,
  title = {{Do NOT Classify and Count: Hybrid Attribute Control Success Evaluation}},
  author = {Felix Matthias Saaro, Pius {von Däniken}, Mark Cieliebak and Jan Milan Deriu},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.48},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.48}
}

References

36 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dhananjay Ashok and Barnabas Poczos. 2024. Controllable text generation in the instruction-tuning era. Preprint, arXiv:2405.01490. — [arXiv:2405.01490](https://arxiv.org/abs/2405.01490) find
  2. Antonio Bella, Cesar Ferri, José Hernández-Orallo, and María José Ramírez-Quintana. 2010. Quantification via probability estimators. In 2010 IEEE International Conference on Data Mining, pages 737–742. find
  3. Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul A. Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep universal probabilistic programming. J. Mach. Learn. Res., 20:28:1–28:6. find
  4. Lital Binyamin, Yoad Tewel, Hilit Segev, Eran Hirsch, Royi Rassin, and Gal Chechik. 2024. Make it count: Text-to-image generation with an accurate number of objects. arXiv preprint arXiv:2406.10210. — [arXiv:2406.10210](https://arxiv.org/abs/2406.10210) find
  5. Black Forest Labs. 2024. Flux. https://github.com/black-forest-labs/flux. ↗
  6. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In Computer Vision – ECCV 2020, pages 213–229, Cham. Springer International Publishing.
  7. Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evalaution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653, Melbourne, Australia. Association for Co… find
  8. Yihan Chen, Benfeng Xu, Quan Wang, Yi Liu, and Zhendong Mao. 2024. Benchmarking large language models on controllable generation under diversified instructions. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17808–17816. find
  9. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations.
  10. Jan Deriu, Pius von Däniken, Don Tuggener, and Mark Cieliebak. 2023. Correction of errors in preference ratings from automated metrics for text generation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6456–6474, Toronto, Canada. Association for Computational Linguist… find
  11. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. 2024. Scaling rectified flow transformers for high-resolution image synthesis. In Proceeding… find
  12. George Forman. 2008. Quantifying counts and costs via classification. Data Mining and Knowledge Discovery, 17:164–206. find
  13. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguisti…
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  15. Sebastian Hartwig, Dominik Engel, Leon Sick, Hannah Kniesel, Tristan Payer, Poonam Poonam, Michael Glöckler, Alex Bäuerle, and Timo Ropinski. 2025. A survey on quality metrics for text-to-image generation. Preprint, arXiv:2403.11821. — [arXiv:2403.11821](https://arxiv.org/abs/2403.11821) find
  16. Matthew D. Hoffman and Andrew Gelman. 2014. The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo. Journal of Machine Learning Research, 15(47):1593–1623. find
  17. Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. 2023. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303.11897. find
  18. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  19. Pei Ke, Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Xiaoyan Zhu, and Minlie Huang. 2022. CTRLEval: An unsupervised reference-free metric for evaluating controlled text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2… find
  20. Muhammad Khalifa, Hady Elsahar, and Marc Dymetman. 2021. A distributional approach to controlled text generation. In International Conference on Learning Representations. find
  21. Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021. GeDi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4929–4952, Punta Cana, Do…
  22. Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. 2024. Imagen-hub: Standardizing the evaluation of conditional image generation models. In ICLR. find
  23. Xun Liang, Hanyu Wang, Yezhaohui Wang, Shichao Song, Jiawei Yang, Simin Niu, Jie Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, and Zhiyu Li. 2024. Controllable text generation for large language models: A survey. Preprint, arXiv:2408.12599.
  24. Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. 2024. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.01291. find
  25. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc.
  26. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01696. find
  27. Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. 2016. SemEval-2016 task 4: Sentiment analysis in Twitter. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1–18, San Diego, California. Association for Computational L…
  28. Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Marc Aubreville. 2024. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In The Twelfth International Conference on Learning Representations. find
  29. Du Phan, Neeraj Pradhan, and Martin Jankowiak. 2019. Composable effects for flexible and accelerated probabilistic programming in numpyro. arXiv preprint arXiv:1912.11554. — [arXiv:1912.11554](https://arxiv.org/abs/1912.11554) find
  30. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3.
  31. Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  32. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695. find
  33. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502– 518, Vancouver, Canada. Association for Computational Linguistics.
  34. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. find
  35. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  36. Pius von Däniken, Jan Deriu, Don Tuggener, and Mark Cieliebak. 2022. On the effectiveness of automated metrics for text generation systems. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1503–1522, Abu Dhabi, United Arab Emirates. Association for Computational Lingui… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saaro-2026-do-not-classify-count-hybrid/