Elamur Research⌕
☰
home / papers / 2026 / BIS Reasoning 1.0: The First Large-Scale Japanese Benchmark for Belief-Inconsistent Syllogistic Reasoning

BIS Reasoning 1.0: The First Large-Scale Japanese Benchmark for Belief-Inconsistent Syllogistic Reasoning

Summary

Large language models often show belief bias: they judge an argument by whether its conclusion sounds plausible rather than by whether it follows logically. Japanese evaluation resources have not addressed this well. JFLD relies on synthetic, unnatural vocabulary, and NeuBAROCO is small and only partly focused on conflicts with common beliefs. The authors fill this gap with BIS Reasoning 1.0, a set of 5,000 Japanese syllogisms. Each item has two premises and a conclusion that is formally valid but runs against commonsense or factual knowledge. Models answer zero-shot whether the conclusion follows, and the correct answer is always "Yes". Models built for explicit reasoning almost saturate the benchmark. GPT-4o and older Japanese-specialized models do much worse, while a recent llm-jp release closes much of that gap. The authors conclude that dedicated reasoning optimization matters more for resisting belief bias than language specialization or model size. They also find that accuracy depends on prompt wording and inference-time reasoning effort. This matters for law, medicine and science, where strict logic must take precedence over intuition.

Key points

  • Dataset: 5,000 valid but belief-conflicting syllogisms in natural Japanese, written by native or highly proficient annotators. Topics span 46 raw semantic categories, merged into 10 final ones (e.g., animals & living things, logic & structure, technology, society).
  • Quality control: the authors first reviewed a 10% sample manually and fed back corrections, then reviewed the full set. They targeted syntactic errors, duplicates, ambiguous premises and category imbalance.
  • Evaluation protocol: a binary Yes/No entailment judgment from Japanese prompts. Answers were extracted by heuristic pattern matching, with Qwen3-32B acting as LLM-as-a-judge when responses were long.
  • Results:
    • Qwen3-32B: about 99%
    • GPT-5-mini: up to about 99.7%
    • GPT-4o: around 80%
    • Earlier Japanese LLMs: often below 60%
    • llm-jp-3.1-13b-instruct4: mid-80s
  • Comparison with prior resources: NeuBAROCO's Japanese portion has fewer than 800 NLI items and fewer than 100 multiple-choice items. BIS Reasoning 1.0 is much larger and focuses exclusively on belief-inconsistent cases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4p65gcdbomon

@inproceedings{nguyen2026bis,
  title = {{BIS Reasoning 1.0: The First Large-Scale Japanese Benchmark for Belief-Inconsistent Syllogistic Reasoning}},
  author = {Nguyen, Ha Thanh and Tachibana, Hideyuki and Liu, Chaoran and Liu, Qianying and Noe, Su Myat and Takeda, Koichi and Kurohashi, Sadao},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4p65gcdbomon},
  url = {https://doi.org/10.63317/4p65gcdbomon}
}

References

19 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akiko Aizawa, Eiji Aramaki, Bowen Chen, Fei Cheng, Hiroyuki Deguchi, Rintaro Enomoto, Kazuki Fujii, Kensuke Fukumoto, Takuya Fukushima, Namgi Han, et al. 2024. Llm-jp: A cross-organizational project for the research and development of fully open japanese llms. arXiv preprint arXiv:2407.03963. — [arXiv:2407.03963](https://arxiv.org/abs/2407.03963) find
  2. Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, and Mitsuhiro Okada. 2023. Evaluating large language models with NeuBAROCO: Syllogistic reasoning ability and human-like biases. In Proceedings of the 4th Natural Logic Meets Machine Learning Workshop, pages 1–11, Nancy, France. Associat… find
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find
  4. Prasoon Bajpai, Niladri Chatterjee, Subhabrata Dutta, and Tanmoy Chakraborty. 2024. Can llms replace neil degrasse tyson? evaluating the reliability of llms as science communicators. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15895–15912. find
  5. Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022. Language models show human-like content effects on reasoning tasks. arXiv preprint arXiv:2207.07051. find
  6. J St BT Evans, Julie L Barston, and Paul Pollard. 1983. On the conflict between logic and belief in syllogistic reasoning. Memory & cognition, 11(3):295–306. find
  7. Jörg Frohberg and Frank Binder. 2022. CRASS: A novel data set and benchmark to test counterfactual reasoning of large language models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2126–2140, Marseille, France. European Language Resources Association. find
  8. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213. find our card
  9. Jiaxuan Li, Lang Yu, and Allyson Ettinger. 2023. Counterfactual reasoning: Testing language models’ understanding of hypothetical scenarios. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 804–815, Toronto, Canada. Associatio… find
  10. Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. 2023. Logiqa 2.0-an improved dataset for logical reasoning in natural language understanding. IEEE ACM Trans. Audio Speech Lang. Process. find
  11. Terufumi Morishita, Atsuki Yamaguchi, Gaku Morio, Hikaru Tomonari, Osamu Imaichi, and Yasuhiro Sogawa. 2024. JFLD: A Japanese benchmark for deductive reasoning based on formal logic. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evalua… find
  12. Ha Thanh Nguyen, Randy Goebel, Francesca Toni, Kostas Stathis, and Ken Satoh. 2023. A negation detection assessment of gpts: analysis with the xnot360 dataset. arXiv preprint arXiv:2306.16638. — [arXiv:2306.16638](https://arxiv.org/abs/2306.16638) find
  13. Kentaro Ozeki, Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, and Mitsuhiro Okada. 2024. Exploring reasoning biases in large language models through syllogism: Insights from the neubaroco dataset. In Findings of the Association for Computational Linguistics ACL 2024, pages 16063–1607… find
  14. Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. 2019. Counterfactual story reasoning and generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lang… find
  15. Tomoki Sugimoto, Yasumasa Onoe, and Hitomi Yanaka. 2023. Jamp: Controlled Japanese temporal inference dataset for evaluating generalization capacity of language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), p… find
  16. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837. find our card
  17. Chenwei Yan, Xiangling Fu, Yuxuan Xiong, Tianyi Wang, Siu Cheung Hui, Ji Wu, and Xien Liu. 2025. LLM sensitivity evaluation framework for clinical diagnosis. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3083–3094, Abu Dhabi, UAE. Association for Computation… find
  18. Hitomi Yanaka and Koji Mineshima. 2021. Assessing the generalization capacity of pre-trained language models through japanese adversarial natural language inference. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 337–349. find
  19. Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2025. Reclor: A reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2026-bis-reasoning-1-0-first/