Elamur Research⌕
☰
home / papers / 2025 / WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More Challenging

WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More Challenging

Paper · 2025 · Ahmed Elhady, Eneko Agirre and Mikel Artetxe

Summary

Multiple-choice benchmarks for LLMs saturate quickly. Building harder replacements, or writing extra plausible distractors as MMLU-Pro does, is expensive and usually needs human checking. The authors propose WiCkeD (Wild-Card Distractor), a fully automatic transformation. For each question, one existing option is picked uniformly at random and swapped for "None of the above". If the removed option was the gold answer, the wildcard becomes the correct choice. This would break questions where several options are partly right and only the best one counts (Single Best Answer, SBA). Such items are detected with a BERT classifier trained on GPT-4o-mini labels and left unchanged. The authors applied WiCkeD to 7 benchmarks through Eval-Harness and tested 18 open-weight LLMs. Accuracy fell by 12.1 points on average, and model rankings shifted. The drop remains under chain-of-thought prompting and also appears for closed models. The method offers a cheap way to revive saturated test sets and to check whether models notice when no listed option is right.

Key points

  • Benchmarks: MMLU, MMLU-Pro, MMLU-Redux, GPQA, CommonsenseQA, TruthfulQA and Arc-challenge. Five random WiCkeD variants were built per benchmark, and results are reported as mean ± std.
  • SBA filter: 4,000 examples were labelled by GPT-4o-mini, and the evaluation split was also checked manually. The resulting BERT classifier (ahmedselhady/bert-base-uncased-sba-clf) reaches over 98.9% recall on SBA items. About 20% of MMLU-family items are SBA, compared with under 5% elsewhere.
  • Direct-prompting results: Drops range from 7.2 to 19.7 points. Qwen2.5-7B loses the most (about 19.7), while DeepSeek-R1-distilled Qwen 7B loses the least (about 7.3). The degradation does not shrink as models get larger.
  • Chain-of-thought results: With zero-shot CoT on MMLU, MMLU-Pro, MMLU-Redux and GPQA, drops still range from 1.4 to 14.6 points. Instruction-tuned and R1-distilled models degrade least, at roughly 2% or less. Closed models (GPT-4o, GPT-4o-mini, Gemini Flash 2.0) lose 3.9–8.2 points.
  • Release: Code and data are available at github.com/ahmedselhady/wicked-benchmarks. Evaluation used 5-shot prompting so that the few-shot examples usually include a case where "None of the above" is correct.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of the Basque CountryLeioa · Spain57 works · 16,520 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Ahmed Elhady, Eneko Agirre and Mikel Artetxe. 2025. WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More Challenging. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-short.94

@inproceedings{elhady2025wicked,
  title = {{WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More Challenging}},
  author = {Ahmed Elhady, Eneko Agirre and Mikel Artetxe},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-short.94},
  url = {https://doi.org/10.18653/v1/2025.acl-short.94}
}

References

33 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri. 2024. Medexpqa: Multilingual benchmarking of large language models for medical question answering. Artificial Intelligence in Medicine, 155:102938. find
  2. Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. 2024. Artifacts or abduction: How do llms answer multiple-choice questions without the question? Preprint, arXiv:2402.12483.
  3. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.
  4. Jo-Anne Sinnige-Egger David DiBattista and Glenda Fortuna. 2014. The “none of the above” option in multiple-choice testing: An experimental study. The Journal of Experimental Education, 82(2):168–183. find
  5. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in ll…
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805.
  7. Filip Dochy, George Moerkerke, Erik De Corte, and Mien Segers. 2001. The assessment of quantitative problem-solving skills with “none of the above”-items (nota items). European Journal of Psychology of Education, 16:163–177. find
  8. Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12039–12050, Bangkok, Thailand. A… find
  9. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  10. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. 2024. Are we done wit… find
  11. Shahriar Golchin and Mihai Surdeanu. 2024. Time travel in LLMs: Tracing data contamination in large language models. In The Twelfth International Conference on Learning Representations.
  12. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.
  14. Pedram Hosseini, Jessica M. Sin, Bing Ren, Bryceton G. Thomas, Elnaz Nouri, Ali Farahanchi, and Saeed Hassanpour. 2024. A benchmark for long-form medical question answering. Preprint, arXiv:2411.09834. — [arXiv:2411.09834](https://arxiv.org/abs/2411.09834) find
  15. Yunjie Ji, Liangyu Chen, Chenxiao Dou, Baochang Ma, and Xiangang Li. 2022. To answer or not to answer? improving machine reading comprehension model with span-based contrastive learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1292–1300, Seattle, United States… find
  16. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  17. Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. 2024. Think twice before trusting: Self-detection for large language models through comprehensive answer reflection. Preprint, arXiv:2403.09972. find
  18. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. Preprint, arXiv:2109.07958.
  19. Jeri L Little. 2023. Does using none-of-the-above (nota) hurt students’ confidence? Journal of Intelligence, 11(8):157. find
  20. Jeri L Little and Elizabeth Ligon Bjork. 2015. Optimizing multiple-choice tests as tools for learning. Memory & cognition, 43:14–26. find
  21. Jeri L Little, Elise A Frickey, and Alexandra K Fung. 2019. The role of retrieval in answering multiple-choice questions. Journal of Experimental Psychology: Learning, Memory, and Cognition, 45(8):1473. find
  22. Timothy R. McIntosh, Teo Susnjak, Nalin Arachchilage, Tong Liu, Paul Watters, and Malka N. Halgamuge. 2024. Inadequacies of large language model benchmarks in the era of generative artificial intelligence. Preprint, arXiv:2402.09880. — [arXiv:2402.09880](https://arxiv.org/abs/2402.09880) find
  23. Aidar Myrzakhan, Sondos Mahmoud Bsharat, and Zhiqiang Shen. 2024. Open-llm-leaderboard: From multi-choice to open-style questions for llms evaluation, benchmark, and arena. Preprint, arXiv:2406.07545. — [arXiv:2406.07545](https://arxiv.org/abs/2406.07545) find
  24. Timothy N Odegard and Joshua D Koen. 2007. “none of the above” as a correct and incorrect alternative on a multiple-choice test: Implications for the testing effect. Memory, 15(8):873–885. find
  25. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  26. Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, Mexico City, Mexico. Association for Computational Linguistics.
  27. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  28. Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, an… find
  29. Joshua Robinson, Christopher Michael Rytting, and David Wingate. 2023. Leveraging large language models for multiple choice question answering. Preprint, arXiv:2210.12353.
  30. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singap…
  31. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  32. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark…
  33. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. Preprint, arXiv:2309.03882. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elhady-2025-wicked-simple-method-make-multiple/