Elamur Research⌕
☰
home / papers / 2026 / Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection

Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection

Paper · 2026 · Zhiwei Liu, Yupeng Cao, Yuechen Jiang et al.

Summary

LLMs are increasingly used to judge financial claims, but existing benchmarks such as FinFact and FinDVer test verification in one fixed setting. They cannot show whether a model's verdict on an identical claim changes with the surrounding context or the language. The authors address this with MFMD-Scen, a benchmark built with financial experts. They took Snopes-derived claims from FinFact, restored the full claim texts and kept items of global relevance. Native speakers reviewed and corrected translations into Chinese, Greek, and Bengali. Each claim is then wrapped in scenarios along three axes: investor role plus personality, role plus market region, and role plus ethnicity and religious belief. Bias is measured as the absolute F1 gap between predictions made with and without the scenario. Across 22 commercial and open-source LLMs, scenario context consistently shifts the decision boundary. The effect is strongest for retail-investor and herding framings and for emerging Asian markets, and it grows in low-resource languages. These findings matter for anyone deploying LLMs in multilingual financial fact-checking.

Key points

  • Three subtasks:
    • MFMD-persona: three roles (retail investor, professional/institutional investor, company owner/manager) combined with five behavioral-finance biases (Overconfidence, Loss Aversion, Herding, Anchoring, Confirmation Bias). Each pairing comes in an explicit and an implicit variant.
    • MFMD-region: different financial markets, such as the USA, Europe, China Mainland, the UAE and Australia.
    • MFMD-identity: ethnicity and faith combined with two individual roles.
  • Task and metric: binary True/False claim verification. Bias is the absolute difference between F1 with the scenario and F1 for the scenario-free baseline.
  • Identity effects interact with role: the same group can flip from negative bias as a retail investor to positive bias as a company owner. This flip is especially marked for Islamic identities. American identities tend to be overestimated and Chinese identities underestimated.
  • Reasoning: explicit reasoning gives unreliable gains for small models and clearer benefits only for large ones.
  • Limitations: the dataset is label-imbalanced, with mostly false claims. Some regions had only two annotators for the human-performance comparison.
  • Code and data: github.com/lzw108/FMD

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States241 works · 54,053 citations🎓The University of MelbourneMelbourne · Australia100 works · 22,618 citations🎓McGill UniversityMontreal · Canada87 works · 11,027 citations🎓University of ManchesterManchester · United Kingdom87 works · 44,938 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada54 works · 13,565 citations🎓Athens University of Economics and BusinessAthens · Greece19 works · 9,652 citations🎓University of FloridaGainesville · United States33 works · 364 citations🎓Islamic University of TechnologyDhaka · Bangladesh10 works · 555 citations🔬Athena Research and Innovation Center In Information Communication & Knowledge TechnologiesMarousi · Greece9 works · 3,461 citations🎓Stevens Institute of TechnologyHoboken · United States10 works · 95 citations🎓Halmstad UniversityHalmstad · Sweden3 works · 37 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Zhiwei Liu, Yupeng Cao, Yuechen Jiang, Mohsinul Kabir, Polydoros Giannouris, Chen Xu, Ziyang Xu, Tianlei Zhu, Md. Tariquzzaman, Triantafillos Papadopoulos, Yan Wang, Lingfei Qian, Xueqing Peng, Zhuohan Xie, Ye Yuan, Saeed Almheiri, Abdulrazzaq Alnajjar, Ming-Bin Chen, Harry Stuart, Paul Thompson, Prayag Tiwari, Alejandro Lopez-Lira, Xue Liu, Jimin Huang and Sophia Ananiadou. 2026. Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.479

@inproceedings{liu2026same,
  title = {{Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection}},
  author = {Zhiwei Liu, Yupeng Cao, Yuechen Jiang, Mohsinul Kabir, Polydoros Giannouris, Chen Xu, Ziyang Xu, Tianlei Zhu, Md. Tariquzzaman, Triantafillos Papadopoulos, Yan Wang, Lingfei Qian, Xueqing Peng, Zhuohan Xie, Ye Yuan, Saeed Almheiri, Abdulrazzaq Alnajjar, Ming-Bin Chen, Harry Stuart, Paul Thompson, Prayag Tiwari, Alejandro Lopez-Lira, Xue Liu, Jimin Huang and Sophia Ananiadou},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.479},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.479}
}

References

46 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Introducing claude sonnet 4.5. Official Vertex AI documentation for Claude Sonnet 4.5. find
  2. Association of Statisticians of American Religious Bodies (ASARB). 2023. 2020 u.s. religion census. https://www.usreligioncensus.org/sites/ default/files/2023-10/2020_US_Religion_ Census.pdf. ↗
  3. Brad M Barber and Terrance Odean. 2001. Boys will be boys: Gender, overconfidence, and common stock investment. The Quarterly Journal of Economics, 116(1):261–292. find
  4. Pietro Bini, Lin William Cong, Xing Huang, and Lawrence J Jin. 2025. Behavioral economics of ai: Llm biases and corrections. Available at SSRN 5213130. find
  5. Yupeng Cao, Haohang Li, Yangyang Yu, and Shashidhar Reddy Javaji. 2025. Capybara at the financial misinformation detection challenge task: chain-of-thought enhanced financial misinformation detection. In Proceedings of the Joint Workshop of the 9th Financial Technology and Natural Language Processin… find
  6. Census of India 2011. 2021. Distribution of population by religions. https://censusindia.gov.in/ nada/index.php/catalog/40443/download/ 44077/DROP_IN_ARTICLE-04.pdf. ↗
  7. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  9. Jessica Maria Echterhoff, Yao Liu, Abeer Alessa, Julian McAuley, and Zexue He. 2024. Cognitive bias in decision-making with llms. In Findings of the association for computational linguistics: EMNLP 2024, pages 12640–12653. find
  10. Dinesh Gabhane, A Sharma, and Rupam Mukherjee. 2023. Behavioral finance: exploring the influence of cognitive biases on investment decisions. Boletin de Literatura Oral-The Literary Journal, 10(1):3133– 3141. find
  11. Thomas Gilovich, Dale Griffin, and Daniel Kahneman. 2002. Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge University Press. find
  12. Patrick Haller, Jannis Vamvas, Rico Sennrich, and Lena Ann Jäger. 2025. Leveraging in-context learning for political bias testing of llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24718–24738. find
  13. Md Zobaer Hossain, Md Ashraful Rahman, Md Saiful Islam, and Sudipta Kar. 2020. Banfakenews: A dataset for detecting fake news in bangla. arXiv preprint arXiv:2004.08789. find
  14. Xuming Hu, Zhijiang Guo, GuanYu Wu, Aiwei Liu, Lijie Wen, and Philip S Yu. 2022. Chef: A pilot chinese dataset for evidence-based fact-checking. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 33… find
  15. Kung-Hsiang Huang, Hou Pong Chan, Kathleen McKeown, and Heng Ji. 2025. Manitweet: A new benchmark for identifying manipulation of news on social media. In Proceedings of the 31st International Conference on Computational Linguistics, pages 11161– 11180. find
  16. International Monetary Fund. 2025a. Regional economic outlook. find
  17. International Monetary Fund. 2025b. Regional economic outlook for asia and pacific. find
  18. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  19. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. find
  20. Daniel Kahneman and Amos Tversky. 1979. Prospect theory: An analysis of decision under risk. Econometrica, 47(2):263–292. find
  21. Mahammed Kamruzzaman, Md Minul Islam Shovon, and Gene Kim. 2023. Banmani: A dataset to identify manipulated social media news in bangla. In Proceedings of the Workshop on Computational Terminology in NLP and Translation Studies (ConTeNTS) Incorporating the 16th Workshop on Building and Using Compara… find
  22. Haein Kong, Yongsu Ahn, Sangyub Lee, and Yunho Maeng. 2024. Gender bias in llm-generated interview responses. arXiv preprint arXiv:2410.20739. — [arXiv:2410.20739](https://arxiv.org/abs/2410.20739) find
  23. Dongjun Lee and Heesoo Park. 2025. Dunamu ml at the financial misinformation detection challenge task: improving supervised fine-tuning with llm-based data augmentation. In Proceedings of the Joint Workshop of the 9th Financial Technology and Natural Language Processing (FinNLP), the 6th Financial N… find
  24. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025a. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
  25. Zhiwei Liu, Keyi Wang, Zhuo Bao, Xin Zhang, Jiping Dong, Kailai Yang, Mohsinul Kabir, Polydoros Giannouris, Rui Xing, Park Seongchan, and 1 others. 2025b. Finnlp-fnp-llmfinlegal-2025 shared task: financial misinformation detection challenge task. In Proceedings of the Joint Workshop of the 9th Finan… find
  26. Zhiwei Liu, Xin Zhang, Kailai Yang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2025c. Fmdllama: Financial misinformation detection based on large language models. In Companion Proceedings of the ACM on Web Conference 2025, pages 1153– 1157. find
  27. Zheyang Luo, Guangbin Zhang, Jiahao Xiao, Xuankang Zhang, Yulin Dou, and Jiangming Liu. 2025. Fmd-mllama at the financial misinformation detection challenge task: multimodal reasoning and evidence generation. In Proceedings of the Joint Workshop of the 9th Financial Technology and Natural Language P… find
  28. Ulrike Malmendier and Geoffrey Tate. 2005. Ceo overconfidence and corporate policies. Journal of Finance, 60(6):2661–2700. find
  29. Qiong Nan, Juan Cao, Yongchun Zhu, Yanyan Wang, and Jintao Li. 2021. Mdfend: Multi-domain fake news detection. In Proceedings of the 30th ACM international conference on information & knowledge management, pages 3343–3347. find
  30. OpenAI. 2025. Gpt-5 system card. Official OpenAI announcement for the GPT-5 model family. find
  31. OpenAI. 2025. Introducing gpt-4.1 in the api. Official OpenAI announcement for the GPT-4.1. find
  32. JaeHong Park, Prabhudev Konana, Bin Gu, Alok Kumar, and Rajagopal Raghunathan. 2010. Confirmation bias, overconfidence, and investment performance: Evidence from stock message boards. McCombs research paper series no. IROM-07-10. find
  33. Aman Rangapur, Haoran Wang, and Kai Shu. 2023. Investigating online financial misinformation and its consequences: A computational perspective. arXiv preprint arXiv:2309.12363. — [arXiv:2309.12363](https://arxiv.org/abs/2309.12363) find
  34. Rajesh Ranjan, Shailja Gupta, and Surya Narayan Singh. 2024. A comprehensive survey of bias in llms: Current landscape and future directions. arXiv preprint arXiv:2409.16430. — [arXiv:2409.16430](https://arxiv.org/abs/2409.16430) find
  35. Sunil Sharma and Sushil Bikhchandani. 2000. Herd Behavior in Financial Markets: A Review. IMF Working Papers, 00(48):1. find
  36. Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS nexus, 3(9):pgae346. find
  37. Amir Taubenfeld, Yaniv Dover, Roi Reichart, and Ariel Goldstein. 2024. Systematic biases in llm simulations of debates. arXiv preprint arXiv:2402.04049.
  38. Ivo Vlaev, Nick Chater, and Neil Stewart. 2007. Relativistic financial decisions: Context effects on retirement saving and investment risk preferences. Judgment and Decision Making, 2(5):292–311. find
  39. Wikipedia contributors. Christianity in the middle east. https://en.wikipedia.org/wiki/ Christianity_in_the_Middle_East. ↗
  40. World Bank Group. 2022. Global financial development report. find
  41. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564.
  42. Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, and 1 others. 2024. Finben: A holistic financial benchmark for large language models. Advances in Neural Information Processing Systems, 37:95716–95743. find
  43. Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. Pixiu: a large language model, instruction data and evaluation benchmark for finance. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 33469–… find
  44. Pew Research Center. 2021. Population growth and religious composition. https://www. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, pewresearch.org/religion/2021/09/21/ population-growth-and-religious-composition/. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. find
  45. Pew Research Center. 2023. Measuring religion 2025a. Qwen3 technical report. arXiv preprint in china. https://www.pewresearch.org/ arXiv:2505.09388. wp-content/uploads/sites/20/2023/08/PF_ An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, 2023.08.30_religion-china_REPORT.pdf. Bo Zheng, Bowen Yu, Ch… ↗
  46. Yilun Zhao, Yitao Long, Tintin Jiang, Chengye Wang, Weiyuan Chen, Hongjun Liu, Xiangru Tang, Yiming Zhang, Chen Zhao, and Arman Cohan. 2024. Findver: Explainable claim verification over long and hybrid-content financial documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-same-claim-different-judgment-benchmarking/