Elamur Research⌕
☰
home / papers / 2026 / Multilingual Disparities in LLM-Based Safety Judgments: Evidence from Brand Safety Applications

Multilingual Disparities in LLM-Based Safety Judgments: Evidence from Brand Safety Applications

Paper · 2026 · Songjiang Liu, Riley Grossman, Mike Smith et al.

Summary

The paper tests a common assumption about multilingual LLM judges: that the same content gets the same verdict whatever its language. The authors study this in brand safety, where automated ratings decide whether advertisers place ads next to news. That decision affects publisher income and advertiser reputation.

They collected Euronews stories that were professionally translated into several languages. This gave them parallel content without relying on machine translation. Each language version was scored three times by MiniMax-M2.5 on the 15 GARM categories, with a safety-floor verdict and a graded suitability level (0–3). A smaller sample was re-rated with two other models to check robustness.

Disagreement across languages turned out to be very common. A fixed-effects ANOVA showed that language significantly shifts suitability scores even when article, category and run are controlled. With the primary model, some languages were systematically judged more harshly and others more leniently. The direction of these shifts changed with the model, but significant language effects held for all three. The work shows that LLM safety ratings can treat equivalent content unequally, with consequences for content moderation more broadly.

Key points

  • Dataset and resource: 10,467 Euronews articles with 70,041 language versions in 13 languages. English has the most versions (9,666) and Arabic the fewest (2,374). Ukrainian is not covered, though Russian is. The ratings benchmark, including prompts, URLs and reasoning traces, is released at github.com/sl947/langdiff.
  • How common disagreement is: about 25.8% of article–category pairs show any cross-language disagreement. Among pairs where at least one version received a non-zero risk, the figure rises to 96.8%. Even rare categories such as Tobacco/Vaping and Online Piracy reach 85–98% under this condition.
  • Statistical tests: in the ANOVA, the language effect is significant (F = 8.115, p < 0.001), and so is the language × category interaction. Repeated runs show no significant effect (p = 0.374).
  • Direction of the bias in MiniMax-M2.5:
    • Rated more strictly: English, German and French content.
    • Rated more leniently: Polish, Hungarian, Greek, Turkish and Persian content.
    • Robustness checks used gemini-3.1-flash-lite-preview and deepseek-v4-pro; language effects persisted, but directional patterns differed by model.
  • Recommendations:
    • Evaluate systems on aligned multilingual benchmarks, reporting results by language and category.
    • Send high-disagreement cases to human review.
    • Make models and prompts transparent to advertisers and publishers.
    • Let publishers contest ratings.
    • Causal mechanisms (cultural bias vs. data imbalance) are left open.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indiana University BloomingtonBloomington · United States37 works · 3,685 citations🎓New Jersey Institute of TechnologyNewark · United States15 works · 1,333 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksBias, fairness & safety · 517 works

Cite

Cite as Songjiang Liu, Riley Grossman, Mike Smith, Cristian Borcea and Yi Chen. 2026. Multilingual Disparities in LLM-Based Safety Judgments: Evidence from Brand Safety Applications. Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.mellm-1.26

@inproceedings{liu2026multilingual,
  title = {{Multilingual Disparities in LLM-Based Safety Judgments: Evidence from Brand Safety Applications}},
  author = {Songjiang Liu, Riley Grossman, Mike Smith, Cristian Borcea and Yi Chen},
  booktitle = {Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.mellm-1.26},
  url = {https://doi.org/10.18653/v1/2026.mellm-1.26}
}

References

23 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Isaac Chung and Linda Freienthal. 2026. Cross-lingual stability of LLM judges under controlled generation: Evidence from Finno-Ugric languages. In Proceedings of the First Workshop on Multilingual Multicultural Evaluation, pages 133–148, Rabat, Morocco. Association for Computational Linguistics. find
  2. Neil Fasching and Yphtach Lelkes. 2025. Model-dependent moderation: Inconsistencies in hate speech detection across LLM-based systems. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22271–22285, Vienna, Austria. Association for Computational Linguistics. on Empirical M… find
  3. Dev Pragad. 2024. Don’t cancel brand safety – improve it. AdExchanger. find
  4. Allison Schiff. 2025. Ai is helping brand safety break free from blocklists. AdExchanger. find
  5. Michael Smith, Riley Grossman, Antonio Torres-Aguero, Pritam Sen, Cristian Borcea, and Yi Chen. 2026. Inconsistencies in classification of online news articles: A call for common standards in brand safety services. Preprint, arXiv:2601.01303. — [arXiv:2601.01303](https://arxiv.org/abs/2601.01303) find
  6. Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS Nexus, 3(9):pgae346. find
  7. Soheil Feizi, MohammadTaghi Hajiaghayi, Keivan Rezaei, and Suho Shin. 2024. Online advertisements with llms: Opportunities and challenges. Preprint, arXiv:2311.07601. — [arXiv:2311.07601](https://arxiv.org/abs/2311.07601) find
  8. Xiyan Fu and Wei Liu. 2025. How reliable is multilingual LLM-as-a-judge? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 11040–11053, Suzhou, China. Association for Computational Linguistics. find
  9. Jiahui He, Yiluo Wei, and Gareth Tyson. 2026. Enhancing content moderation with llms: A reddit case study on evaluating and refining human decisions. In Proceedings of the ACM Web Conference 2026, WWW ’26, page 9823–9834, New York, NY, USA. Association for Computing Machinery. find
  10. Alice Hunsberger. 2025. How to use llms for content moderation. Musubi. find
  11. Integral Ad Science. 2024. Our brand safety and suitability overview. Published March 6, 2024. find
  12. Ross W. Johnson, Clay Voorhees, and Farnoosh Kho-dakarami. 2023. Is your brand protected? assessing brand safety risks in digital campaigns. Journal of Advertising Research, 63(3):205–221. find
  13. Mahi Kolla, Siddharth Salunkhe, Eshwar Chandrasekharan, and Koustuv Saha. 2024. Llm-mod: Can large language models assist content moderation? In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA ’24, New York, NY, USA. Association for Computing Machinery. find
  14. Adi Levi, Or Levi, Sardhendu Mishra, and Jonathan Morra. 2025. Ai vs. human moderators: A comparative evaluation of multimodal llms in content moderation for brand safety. Preprint, arXiv:2508.05527. — [arXiv:2508.05527](https://arxiv.org/abs/2508.05527) find
  15. Marco Lewis. 2022. The netzdg and the avia law: How two different legal systems created two different outcomes from similar laws. Wis. Int’l LJ, 40:491. find
  16. Bibek Upadhayay and Vahid Behzadan. 2025. X-guard: Multilingual guard agent for content moderation. In Proceedings of the First Workshop on LLM Security (LLMSEC), pages 54–86, Vienna, Austria. Association for Computational Linguistics. find
  17. Yash Vekaria, Rishab Nithyanand, and Zubair Shafiq. 2024. The inventory is dark and full of misinformation: Understanding ad inventory pooling in the ad-tech supply chain. In 2024 IEEE Symposium on Security and Privacy (SP), pages 1590–1608. find
  18. Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael Lyu. 2024. All languages matter: On the multilingual safety of LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5865– 5877, Bangkok, Thailand. Association for Computatio… find
  19. World Federation of Advertisers. 2024. Wfa discontinues garm. Published August 9, 2024. find
  20. World Population Review. 2026. Alcohol consumption by country 2026. Accessed: 05-01-2026. find
  21. Zheng Xin Yong, Beyza Ermis, Marzieh Fadaee, Stephen Bach, and Julia Kreutzer. 2025. The state of multilingual LLM safety research: From measuring the language gap to mitigating it. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15845–15860, Suzhou,…
  22. ZEFR. 2026. Zefr is the first third-party partner to receive mrc accreditation for content-level brand safety and suitability reporting on youtube. ZEFR Press Release. find
  23. Sarah Masud, Sahajpreet Singh, Viktor Hangya, Alexander Fraser, and Tanmoy Chakraborty. 2024. Hate personified: Investigating the role of LLMs in content moderation. In Proceedings of the 2024 Conference find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-multilingual-disparities-llm-based-safety/