Elamur Research⌕
☰
home / papers / 2026 / Evaluating the Effect of Question Wording Variations on Answer Consistency in Large Language Models

Evaluating the Effect of Question Wording Variations on Answer Consistency in Large Language Models

Summary

LLMs can give different answers to questions that mean the same thing but are phrased differently. This undermines reliability and can make models overly agreeable toward users. The authors study this problem with binary Yes/No questions, which make consistency easy to measure. They turn two classification tasks, commonsense morality judgment and tweet sentiment polarity, into question templates in Japanese and English. Each template is then rewritten in four ways: synonym swaps, antonym swaps that should flip the expected answer, tag-question style phrasing that invites agreement, and a more insistent version of that phrasing. Several open-source and commercial models answer both the original and rewritten forms, and the paired answers are compared. Most models drift toward "Yes" when the wording invites agreement, and antonym rewrites cause the most inconsistency. Some of these biases already appear in pretrained models and persist after post-training. Making models larger does not remove the problem, while few-shot examples with varied phrasings help somewhat. The framework gives developers a direct way to expose wording-driven biases before deployment.

Key points

  • Datasets: JCommonsenseMorality (Japanese) and ETHICS (English) are used for morality QA. WRIME (objective labels, scale −2 to 2) and SemEval-2018 Task 1 valence annotations (scale −3 to 3) are used for sentiment, with neutral items dropped.
  • Templates: Each paraphrase type has 4–5 templates, and one is picked at random per question. In Japanese, agreement-seeking is expressed by changing ですか？ to ですよね？.
  • Metrics:
    • Consistency: how often the answer stays the same, with labels inverted for the antonym case.
    • DiffYes: the shift in the share of "Yes" answers after rewording.
    • DiffAcc: the change in accuracy after rewording.
    • Overall: the harmonic mean of original accuracy, paraphrased accuracy, and Consistency.
  • Main findings: Agreement-seeking phrasing pushes models toward "Yes", and antonym substitution is the weakest spot. Each model also shows its own pattern of sensitivity.
  • Correlations: Wording consistency partly tracks general reasoning ability. It is weakly negatively correlated with emotional-alignment capability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4k8j56pzchi7

@inproceedings{takayama2026evaluating,
  title = {{Evaluating the Effect of Question Wording Variations on Answer Consistency in Large Language Models}},
  author = {Takayama, Junya and Ohagi, Masaya and Mizumoto, Tomoya and Yoshikawa, Katsumasa},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4k8j56pzchi7},
  url = {https://doi.org/10.63317/4k8j56pzchi7}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  2. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… find our card
  3. Junbo Fu, Guoshuai Zhao, Yimin Deng, Yunqi Mi, and Xueming Qian. 2024. Learning to paraphrase for alignment with LLM preference. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2394–2407. find
  4. Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual pre-training for cross-lingual llm adaptation: Enhancing japanese language capabilities. In Proceedings of the First Conference on Langua… find our card
  5. Gemma Team. 2025. Gemma 3 technical report. find
  6. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning ai with shared human values. In Proceedings of the International Conference on Learning Representations (ICLR). find our card
  7. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  8. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  9. Tomoyuki Kajiwara, Chenhui Chu, Noriko Takemura, Yuta Nakashima, and Hajime Nagahara. 2021. WRIME: A new dataset for emotional intensity estimation with subjective and objective annotations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… find
  10. Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. 2024. Can multiple-choice questions really be useful in detecting the abilities of LLMs? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), p… find
  11. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let's verify step by step. In Proceedings of the International Conference on Representation Learning (ICLR), volume 2024, pages 39578–39601. find our card
  12. Ruixi Lin and Hwee Tou Ng. 2023. Mind the biases: Quantifying cognitive biases in language model prompting. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5269–5281. find
  13. Riccardo Lunardi, Vincenzo Della Mea, Stefano Mizzaro, and Kevin Roitero. 2025. On robustness and reliability of benchmark-based evaluation of llms. In 28th European Conference on Artificial Intelligence (ECAI). find
  14. Youmi Ma, Sakae Mizuki, Kazuki Fujii, Taishi Nakamura, Masanari Ohi, Hinari Shimada, Taihei Shiotani, Koshiro Saito, Koki Maeda, Kakeru Hattori, Takumi Okamoto, Shigeki Ishida, Rio Yokota, Hiroya Takamura, and Naoaki Okazaki. 2025. Building instruction-tuning datasets from human-written instructions… find
  15. Meta. 2024. The llama 3 herd of models. find
  16. Microsoft Research. 2024. Phi-4 technical report. find
  17. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 task 1: Affect in tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval), pages 1–17. find our card
  18. Nvidia. 2024. Nemotron-4 340b technical report. find our card
  19. Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Rio Yokota, and Sakae Mizuki. 2024. Building a large japanese web corpus for large language models. In Proceedings of the First Conference on Language Modeling, COLM, page (to appear)… find
  20. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. find our card
  21. Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017. find our card
  22. Qwen team. 2025a. Qwen2.5 technical report. find
  23. Qwen team. 2025b. Qwen3 technical report. find
  24. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In Proceedings of the First Conference on Language Modeling (COLM). find our card
  25. Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Tatia Lee, Rada Mihalcea, and Minlie Huang. 2024. EmoBench: Evaluating the emotional intelligence of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (… find
  26. Masashi Takeshita and Rafal Rzepka. 2025. Jethics: Japanese ethics understanding evaluation dataset. find
  27. Hiroo Takizawa, Saku Sugawara, and Akiko Aizawa. 2025. MCQFormatBench: Robustness tests for multiple-choice questions. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM 2 ), pages 824–846. find
  28. Jan Philip Wahle, Terry Ruas, Yang Xu, and Bela Gipp. 2024. Paraphrase types elicit prompt engineering capabilities. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 11004– 11033. find
  29. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS)… find
  30. Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. Prompt consistency for zero-shot task generalization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2613– 2626. find
  31. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. find our card
  32. Yongshuo Zong, Tingyang Yu, Ruchika Chavhan, Bingchen Zhao, and Timothy M. Hospedales. 2024. Fool your (vision and) language model with embarrassingly simple permutations. In Proceedings of 41st International Conference on Machine Learning (ICML). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/takayama-2026-evaluating-effect-question-wording-variations/