Elamur Research⌕
☰
home / papers / 2026 / An Evaluation of Classifiers for Mapping Generative LLM Responses to Answer Options of Multiple-choice Questionnaires

An Evaluation of Classifiers for Mapping Generative LLM Responses to Answer Options of Multiple-choice Questionnaires

Paper · 2026 · Alisea Stroligo, Anna Shamray, Julian Schelb et al.

Summary

Researchers often have generative LLMs fill in closed-ended questionnaires, for example in persona simulation, machine psychology or benchmark evaluation. The free-form replies then have to be turned back into one of the fixed answer options. This is hard because models add explanations, refuse to answer, or mention several options. The authors compare small, cheap classifiers for this cleaning step.

They tested four neural designs built on fine-tuned BERT-family encoders. Each design scores every pair of answer option and LLM reply. An extra [None] outcome covers invalid replies, detected either by an entropy threshold or by a dedicated class. The baseline is a simple string-matching rule. Evaluation used a new manually annotated set of replies from several state-of-the-art LLMs, drawn from three sources whose answer options differ in complexity.

Unexpectedly, no fine-tuned model beat the rule-based baseline, even on the harder option formats. The best neural model only matched the baseline on one dataset. For standard multiple-choice settings, plain pattern matching therefore remains a competitive option that costs far less compute.

Key points

  • Datasets: questions come from the Regulatory Focus Questionnaire (RFQ, Likert-style), ARC and MMLU-Pro. The manually annotated LLM responses, together with the code, are released on GitHub.
  • Rule-based classifier (RbC): it lower-cases the reply, strips punctuation and splits options into tokens such as "3" and "sometimes". Each option gets a fractional match score capped at 1, and the highest score wins. No match or a tie yields [None].
  • Neural variants:
    • Three entropy-based 1-vs-all models, including EM-O, which is adapted from Schelb et al. (2025).
    • NEM, which uses an explicit [None] class.
  • Main result: none of the variants outperformed RbC, despite more than 35,000 training samples. DistilRoBERTa only equalled RbC accuracy on RFQ, and RoBERTa offered the best cost-performance among the encoders.
  • Handling invalid replies: the entropy-based models had trouble recognising [None] replies on ARC and MMLU-Pro. NEM handled them much better, especially on MMLU-Pro, which has the most invalid replies. Smaller answering LLMs also produced noisier outputs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of KonstanzKonstanz · Germany17 works · 232 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Alisea Stroligo, Anna Shamray, Julian Schelb and Andreas Spitz. 2026. An Evaluation of Classifiers for Mapping Generative LLM Responses to Answer Options of Multiple-choice Questionnaires. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-srw.64

@inproceedings{stroligo2026evaluation,
  title = {{An Evaluation of Classifiers for Mapping Generative LLM Responses to Answer Options of Multiple-choice Questionnaires}},
  author = {Alisea Stroligo, Anna Shamray, Julian Schelb and Andreas Spitz},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.64},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.64}
}

References

46 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gati V. Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, ICML, pages 337–371. find
  2. Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351.
  3. Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120. find
  4. Bojana Bodroža, Bojana M Dinić, and Ljubiša Bojić. 2024. Personality testing of large language models: limited temporal stability, but highlighted prosociality. Royal Society Open Science, 11(10):240180. find
  5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. CoRR, abs/1803.05457.
  6. Washington Cunha, Leonardo Rocha, and Marcos André Gonçalves. 2025. A thorough benchmark of automatic text classification: From traditional approaches to large language models. CoRR, abs/2504.01930. — [arXiv:2504.01930](https://arxiv.org/abs/2504.01930) find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and et al. 2024. The Llama 3 herd of models. CoRR, abs/2407.21783.
  9. John Fields, Kevin Chovanec, and Praveen Madiraju. 2024. A survey of text classification with transformers: How wide? How large? How long? How accurate? How expensive? How safe? IEEE Access, 12:6518–6531. find
  10. Andrea Gasparetto, Matteo Marcuzzo, Alessandro Zangari, and Andrea Albarelli. 2022. A survey on text classification algorithms: From text to predictions. Inf., 13(2):83. find
  11. Ahmad Ghazal, Tilmann Rabl, Minqing Hu, Francois Raab, Meikel Poess, Alain Crolotte, and Hans-Arno Jacobsen. 2013. BigBench: Towards an industry standard benchmark for big data analytics. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, SIGMOD ’13, page 1197–1208… find
  12. Hyukjun Gweon and Matthias Schonlau. 2023. Automated classification for open-ended questions with BERT. Journal of Survey Statistics and Methodology, 12(2):493–504. find
  13. Thilo Hagendorff, Ishita Dasgupta, Marcel Binz, Stephanie C. Y. Chan, Andrew Lampinen, Jane X. Wang, Zeynep Akata, and Eric Schulz. 2024. Machine psychology. Preprint, arXiv:2303.13988. — [arXiv:2303.13988](https://arxiv.org/abs/2303.13988) find
  14. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  15. Rohit Kumar Kaliyar, Anurag Goswami, and Pratik Narang. 2021. FakeBERT: Fake news detection in social media with a BERT-based deep learning approach. Multim. Tools Appl., 80(8):11765–11788. find
  16. Arina Kostina, Marios D. Dikaiakos, Dimosthenis Stefanidis, and George Pallis. 2025. Large language models for text classification: Case study and comprehensive review. CoRR, abs/2501.08457. — [arXiv:2501.08457](https://arxiv.org/abs/2501.08457) find
  17. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevi…
  18. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2025. From generation to judgment: Opportunities and challenges of LLM-as-a-judge. In Proceedings of the 2025 Conference o… find
  19. Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S. Yu, and Lifang He. 2022. A survey on text classification: From traditional to deep learning. ACM Trans. Intell. Syst. Technol., 13(2). find
  20. Xingxuan Li, Yutong Li, Lin Qiu, Shafiq Joty, and Lidong Bing. 2024. Evaluating psychological safety of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1826–1843, Miami, Florida, USA. Association for Computational Linguistics. find
  21. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. CoRR, abs/2009.03300.
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  23. E. Tory Higgins, Ronald S. Friedman, Robert E. Harlow, Lorraine Chen Idson, Ozlem N. Ayduk, and Amy Taylor. 2001. Achievement orientations from subjective histories of success: Promotion pride versus prevention pride. European Journal of Social Psychology, 31(1):3–23. find
  24. Xinyi Lu and Xu Wang. 2024. Generative students: Using LLM-simulated student profiles to support question item evaluation. In Proceedings of the Eleventh ACM Conference on Learning @ Scale, L@S ’24, page 16–27, New York, NY, USA. Association for Computing Machinery. find
  25. John J Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? Working Paper 31122, National Bureau of Economic Research. find
  26. Yang Lu, Jordan Yu, and Shou-Hsuan Stephen Huang. 2023. Illuminating the black box: A psychometric investigation into the multifaceted nature of large language models. Preprint, arXiv:2312.14202. — [arXiv:2312.14202](https://arxiv.org/abs/2312.14202) find
  27. Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2022. Deep learning-based text classification: A comprehensive review. ACM Comput. Surv., 54(3):62:1–62:40. find
  28. Marilù Miotto, Nicola Rossberg, and Bennett Kleinberg. 2022. Who is GPT-3? An exploration of personality, values and demographics. In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS). find
  29. Max Pellert, Clemens M Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. 2024. AI psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. Perspectives on Psychological Science, 19(5):808–826. find
  30. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
  31. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108. find
  32. Julian Schelb, Orr Borin, David Garcia, and Andreas Spitz. 2025. R.u.psycho? Robust unified psychometric testing of language models. Preprint, arXiv:2503.10229. — [arXiv:2503.10229](https://arxiv.org/abs/2503.10229) find
  33. Matthias Schonlau, Julia Weiß, and Jan Marquardt. 2023. Multi-label classification of open-ended questions with BERT. CoRR, abs/2304.02945. find
  34. Gregory Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2025. A psychometric framework for evaluating and shaping personality traits in large language models. Nature Machine Intelligence, 7(12):1954– 1968. find
  35. S. S. Stevens. 1946. On the theory of scales of measurement. Science, 103(2684):677–680. find
  36. Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023. Text classification via large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8990–9005, Singapore. Association for Computational Linguistics.
  37. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  38. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179… find
  39. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, and et al. 2023. Zephyr: Direct distillation of LM alignment. CoRR, abs/2310.16944. find
  40. Sowmya Vajjala and Shwetali Shimangaud. 2025. Text classification in the LLM era - Where do we stand? CoRR, abs/2502.11830. — [arXiv:2502.11830](https://arxiv.org/abs/2502.11830) find
  41. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
  42. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024a. "My Answer is C": First-token probabilities do not match text answers in instruction-tuned language models. In Findings of the Association for Computational Linguistics, ACL.
  43. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024b. MMLU-Pro: A more robust and challenging multi-task language understanding benchmar…
  44. Hanzi Xu, Renze Lou, Jiangshu Du, Vahid Mahzoon, Elmira Talebianaraki, Zhuoan Zhou, Elizabeth Garrison, Slobodan Vucetic, and Wenpeng Yin. 2024. LLMs’ classification performance is overclaimed. CoRR, abs/2406.16203. — [arXiv:2406.16203](https://arxiv.org/abs/2406.16203) find
  45. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, and et al. 2024. Qwen2.5 technical report. CoRR, abs/2412.15115. find
  46. Yazhou Zhang, Mengyao Wang, Qiuchi Li, Prayag Tiwari, and Jing Qin. 2025. Pushing the limit of LLM capacity for text classification. In Companion Proceedings of the ACM on Web Conference 2025, WWW ’25, page 1524–1528, New York, NY, USA. Association for Computing Machinery. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stroligo-2026-evaluation-classifiers-mapping-generative-llm/