Elamur Research⌕
☰
home / papers / 2026 / A Binary Problem in Binary QA: Diverse LLMs or Diverse Question Interpretations? That Is the Ensembling Question

A Binary Problem in Binary QA: Diverse LLMs or Diverse Question Interpretations? That Is the Ensembling Question

Paper · 2026 · Rafael Rosales and Santiago Miret

Summary

LLM ensembles are meant to improve robustness by combining members that fail in different ways. In practice, independently developed models often make the same mistakes. This paper asks where that diversity should come from in yes/no question answering: from several different LLMs, or from one LLM answering several readings of the same question.

In the interpretation setup, GPT-3.5 first proposes three distinct semantic readings of each question. The model under test then restates and answers each reading, and a majority vote picks the final answer. In the model setup, three different LLMs answer one fixed reading, and their outputs are voted on. Free-text replies are mapped to labels such as yes, no or unclear using keyword and regex extraction heuristics, then scored by accuracy.

Tested with GPT and LLaMA families on BoolQ, StrategyQA and PubMedQA, voting across interpretations reliably outperformed voting across models. Model voting usually landed between the best and worst member. Because interpretation diversity needs only a single model, it offers a cheaper route to dependable ensembles. It also exposes ambiguity in benchmark labels.

Key points

  • Models: gpt-3.5-turbo, gpt-3.5-turbo-instruct, gpt-4, LLaMA-2-7B-Chat, LLaMA-3-8B and LLaMA-3-8B-Instruct. GPT-3.5 alone generated all question interpretations.
  • Data: 3270 BoolQ validation questions, 2290 StrategyQA train questions, and 445 PubMedQA questions (a 50% split of PQA-L yes/no items).
  • Two answer-extraction heuristics are used. "Match First" takes the first yes/no-style keyword in the reply. "Match Pattern" tries regex patterns such as <answer> tags before falling back to keyword matching.
  • In a random BoolQ sample, the authors judged about a third of the questions ambiguous. Their manual answers to the generated interpretations often differed from the dataset's ground truth.
  • Example gain: for gpt-3.5-turbo-instruct on PubMedQA, accuracy rose from 42.02 (direct question) to 99.78 with interpretation voting. The advantage over model diversity was especially large for LLaMA-based ensembles.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Information Systems · 523 works

🧭 Topics 2 Language models & pretraining · 1,343 works Question answering · 242 works

Cite

Cite as Rafael Rosales and Santiago Miret. 2026. A Binary Problem in Binary QA: Diverse LLMs or Diverse Question Interpretations? That Is the Ensembling Question. Proceedings of the Fifteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/43t2yvgid7tw

@inproceedings{rosales2026binary,
  title = {{A Binary Problem in Binary QA: Diverse LLMs or Diverse Question Interpretations? That Is the Ensembling Question}},
  author = {Rafael Rosales and Santiago Miret},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/43t2yvgid7tw},
  url = {https://doi.org/10.63317/43t2yvgid7tw}
}

References

44 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. our card
  2. Algirdas Avizienis and John P. J. Kelly. 1984. Fault tolerance by design diversity: Concepts and experiments. Computer, 17(8):67–80. find
  3. Parishad BehnamGhader, Santiago Miret, and Siva Reddy. 2023. Can retriever-augmented language models reason? the blame game between the retriever and the language model. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 15492–15509. Assoc… find
  4. Leo Breiman. 1996. Bagging predictors. Mach. Learn., 24(2):123–140. our card
  5. Abraham Chan, Niranjhana Narayanan, Arpan Gujarati, Karthik Pattabiraman, and Sathish Gopalakrishnan. 2021. Understanding the resilience of neural network ensembles against faulty training data. In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS), pages 110… find
  6. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:… our card
  7. Jeremy R. Cole, Michael J. Q. Zhang, Daniel Gillick, Julian Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. 2023. Selectively answering ambiguous questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 530–543. Association for Computation… our card
  8. Dimitris Dimitriadis and Grigorios Tsoumakas. 2024. Artificial fine-tuning tasks for yes/no question answering. Natural Language Engineering, 30(1):73–95. find
  9. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. A survey on In-context learning. CoRR, cs.CL/2301.00234v4. our card
  10. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 herd of models. CoRR, abs/2407.21783. find
  11. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361. find
  12. Zhiqiang Gong, Ping Zhong, and Weidong Hu. 2019. Diversity in machine learning. IEEE Access, 7:64323–64350. find
  13. Carmelo Gonzales, Michael Martin Pieler, Kevin Maik Jablonka, and Santiago Miret. 2024. Evaluating chemistry prompts for large-language model fine-tuning. In AI for Accelerated Materials Design-NeurIPS 2024. find
  14. Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey. CoRR, abs/2403.14608v6. our card
  15. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  16. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. LLM-blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165–14178, Toronto, Canada. Assoc… our card
  17. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. 2019. Pub-MedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language P… find
  18. Shauharda Khadka, Estelle Aflalo, Mattias Marder, Avrech Ben-David, Santiago Miret, Shie Mannor, Tamir Hazan, Hanlin Tang, and Somdeb Majumdar. 2021. Optimizing memory placement using evolutionary graph reinforcement learning. In International Conference on Learning Representations. find
  19. Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang. 2023. Tree of clarifications: Answering ambiguous questions with retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 996–1009. Asso… our card
  20. Jakub Lála, Odhran O’Donoghue, Aleksandar Shtedritski, Sam Cox, Samuel G. Rodriques, and Andrew D. White. 2023. PaperQA: Retrieval-augmented generative agent for scientific research. CoRR, abs/2312.07559. find
  21. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Proc… find
  22. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023. Making Large Language Models better reasoners with step-aware verifier. CoRR, cs.CL/2206.02336v3. find
  23. Anne-Laure Ligozat, Brigitte Grau, Anne Vilnat, Isabelle Robba, and Arnaud Grappy. 2007. Towards an automatic validation of answers in question answering. In 19th IEEE International Conference on Tools with Artificial Intelligence(ICTAI 2007), volume 2, pages 444–447. find
  24. Bev Littlewood and Douglas R. Miller. 1989. Conceptual modeling of coincident failures in multi-version software. IEEE Transactions on Software Engineering, 15(12):1596–1614. find
  25. Bo Lv, Chen Tang, Yanan Zhang, Xin Liu, Ping Luo, and Yue Yu. 2024. URG: A unified ranking and generation method for ensembling language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4421–4434, Bangkok, Thailand. Association for Computational Linguistics. find
  26. Meta. 2024. Llama. https://llama.meta.com/. Accessed: 2025-02-14. ↗
  27. Santiago Miret, Vui Seng Chua, Mattias Marder, Mariano Phiellip, Nilesh Jain, and Somdeb Majumdar. 2022. Neuroevolution-enhanced multi-objective optimization for mixed-precision quantization. In GECCO ’22: Genetic and Evolutionary Computation Conference, Boston, Massachusetts, USA, July 9 - 13, 2022… find
  28. OpenAI. 2024. ChatGPT. https://www.openai.com/. Accessed: 2025-02-14. ↗
  29. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… our card
  30. Silviu Pitis, Michael R. Zhang, Andrew Wang, and Jimmy Ba. 2023. Boosted prompt ensembles for large language models. CoRR, abs/2304.05970. — [arXiv:2304.05970](https://arxiv.org/abs/2304.05970) find
  31. Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. 2016. Quality diversity: A new frontier for evolutionary computation. Frontiers Robotics AI, 3:40. find
  32. Mathieu Ravaut, Shafiq Joty, and Nancy Chen. 2022. SummaReranker: A multi-task mixture-of-experts re-ranking framework for abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4504–4524, Dublin, Ireland.… find
  33. Rafael Rosales, Pablo Munoz, and Michael Paulitsch. 2023. Assessing the impact of diversity on the resilience of deep learning ensembles: A comparative study on model architecture, output, activation, and attribution. In IEEE/CVF International Conference on Computer Vision, ICCV 2023 - Workshops, Pa… find
  34. Sara Rosenthal, Mihaela A. Bornea, Avirup Sil, Radu Florian, and J. Scott McCarley. 2021. Do answers to boolean questions need explanations? Yes. CoRR, abs/2112.07772v1. — [arXiv:2112.07772](https://arxiv.org/abs/2112.07772) find
  35. Yu Song, Santiago Miret, Huan Zhang, and Bang Liu. 2023. HoneyBee: Progressive instruction finetuning of large language models for materials science. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5724–5739, Singapore. Association for Computational Linguistics. find
  36. Haitian Sun, William W. Cohen, and Ruslan Salakhutdinov. 2023. Answering ambiguous questions with a database of questions, answers, and revisions. CoRR, abs/2308.08661v1. — [arXiv:2308.08661](https://arxiv.org/abs/2308.08661) find
  37. E. Ke Tang, Ponnuthurai N. Suganthan, and Xin Yao. 2006. An analysis of diversity measures. Mach. Learn., 65(1):247–271. find
  38. Bernadette Tix and Kim Binsted. 2024. Better results through ambiguity resolution: Large language models that ask clarifying questions. In Augmented Cognition - 18th International Conference, AC 2024, Held as Part of the 26th HCI International Conference, HCII Proceedings, Part II, volume 14695 of L… find
  39. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288. find
  40. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR. OpenReview.net. our card
  41. Yuguang Wang, Shanbo Cheng, Liyang Jiang, Jiajun Yang, Wei Chen, Muze Li, Lin Shi, Yanfeng Wang, and Hongtao Yang. 2017. Sogou neural machine translation systems for WMT17. In Proceedings of the Second Conference on Machine Translation, pages 410–415, Copenhagen, Denmark. Association for Computation… find
  42. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems, NeurI… find
  43. Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali,… our card
  44. Seniha Esen Yuksel, Joseph N. Wilson, and Paul D. Gader. 2012. Twenty years of mixture of experts. IEEE Transactions on Neural Networks and Learning Systems, 23(8):1177–1193. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rosales-2026-binary-problem-binary-qa-diverse/