Elamur Research⌕
☰
home / papers / 2026 / BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Stories

BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Stories

Paper · 2026 · Yuxuan Ouyang, Yingfeng Luo, JingBo Zhu et al.

Summary

Most studies of social bias in LLMs are English-only and rely on short-form probes. It is therefore unclear whether the attribute patterns seen in English long-form generation carry over to other languages. To test this, the authors built BIASEDTALES-ML, a parallel collection of about 350k children's stories. Three open-weight models wrote them natively in eight languages. The prompts were localized by native speakers rather than machine-translated. They also cover every combination of nationality, religion, social class, parent role and child gender.

A generator–extractor pipeline uses an LLM to pull out narrative features such as character traits, settings and themes. Distribution-based measures then compare groups, including Jensen–Shannon Divergence and log-odds lexical markers.

The main finding is that English attribute distributions often fail to hold in other languages, and the gap is largest for low-resource Swahili. In that setting, the smaller model writes fewer valid stories and shows stronger divergence. The authors argue this calls for multilingual, distributional evaluation of alignment.

Key points

  • The corpus holds 349,920 stories: 2,916 attribute configurations per language × 8 languages × 3 models × 5 samples. The configurations combine 27 nationalities, 6 religions, 2 classes, 3 parent roles and 3 child genders.
  • The languages are English, Chinese, Japanese and Korean (no grammatical gender), Spanish, Russian and Arabic (grammatical gender), and Swahili (low-resource).
  • The generators are Qwen-3-8B, Llama-3.1-8B and Llama-3.2-1B, run via vLLM at high temperature. Stories that fail automatic language identification are removed.
  • Generation quality (Valid Story Rate) is plotted against bias strength (JSD), and the 1B model diverges most in low-resource settings.
  • The dataset, code and the interactive Biased Tales Explorer are released on Hugging Face Spaces.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northeastern UniversityShenyang · China42 works · 1,940 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Yuxuan Ouyang, Yingfeng Luo, JingBo Zhu and Tong Xiao. 2026. BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Stories. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.862

@inproceedings{ouyang2026biasedtales,
  title = {{BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Stories}},
  author = {Yuxuan Ouyang, Yingfeng Luo, JingBo Zhu and Tong Xiao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.862},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.862}
}

References

29 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. BedtimeStory.ai. 2023. AI Powered Story Creator \| Bedtimestory.ai. find
  3. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. find
  4. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics.
  5. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from lan-17428 9 guage corpora contain human-like biases. Science, 356(6334):183–186. find
  6. Victoria Cooper. 2014. Children’s developing identity. A critical companion to early childhood, pages 281– 296. find
  7. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474.
  8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  9. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam De Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, and 1 others. 2022. Challenges and strategies in cross-cultural nlp. arXiv preprint arXiv:2203.10020.
  10. Nicole Kobie. 2023. AI Is Telling Bedtime Stories to Your Kids Now. Wired. Section: tags. find
  11. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
  12. Jianhua Lin. 2002. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145–151. find
  13. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634.
  14. Li Lucy and David Bamman. 2021. Gender and representation bias in gpt-3 generated stories. In Proceedings of the third workshop on narrative understanding, pages 48–55.
  15. Kristian Lum, Jacy Reese Anthis, Kevin Robinson, Chirag Nagpal, and Alexander Nicholas D’Amour. 2025. Bias in language models: Beyond trick tests and towards ruted evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025… find
  16. Yingfeng Luo, Ziqiang Xu, Yuxuan Ouyang, Murun Yang, Dingyang Lin, Kaiyan Chang, Tong Zheng, Bei Li, Peinan Feng, Quan Du, Tong Xiao, and Jingbo Zhu. 2025. Beyond english: Toward inclusive and scalable multilingual machine translation with llms. CoRR, abs/2511.07003. — [arXiv:2511.07003](https://arxiv.org/abs/2511.07003) find
  17. Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372–403. find
  18. Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456.
  19. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  20. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022. Bbq: A hand-built bias benchmark for question answering. Preprint, arXiv:2110.08193.
  21. Donya Rooein, Amanda Cercas Curry, and Dirk Hovy. 2023. Know your audience: Do LLMs adapt to different age and education levels? arXiv preprint arXiv:2312.02065.
  22. Donya Rooein, Vilém Zouhar, Debora Nozza, and Dirk Hovy. 2025. Biased tales: Cultural and topic bias in generating children’s stories. CoRR, abs/2509.07908. find
  23. Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. 2024. The language barrier: Dissecting safety challenges of llms in multilingual contexts. arXiv preprint arXiv:2401.13136.
  24. Spriha Srivastava. 2023. I use ChatGPT to write stories for my 5-year-old. It’s fun, innovative, and makes bedtime less stressful. find
  25. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  26. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36:80079–80110.
  27. Zheng-Xin Yong, Beyza Ermis, Marzieh Fadaee, Stephen Bach, and Julia Kreutzer. 2025. The state of multilingual llm safety research: From measuring the language gap to mitigating it. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15856–15871.
  28. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623. 17429 10
  29. Category Values Nationality (N = 27) Americas: American, Mexican, Brazilian, Argentine Europe: British, French, German, Spanish, Russian, Ukrainian Asia: Chinese, Japanese, Korean, Indian, Filipino, Indonesian, Thai, Vietnamese, Sri Lankan Middle East: Iranian, Egyptian, Saudi Africa: Nigerian, Ethi… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ouyang-2026-biasedtales-ml-multilingual-dataset-analyzing/