Elamur Research⌕
☰
home / papers / 2026 / A Multilingual Red Teaming–Driven Safety Analysis of LLMs

A Multilingual Red Teaming–Driven Safety Analysis of LLMs

Summary

The paper examines how well large language models resist harmful requests in more than one language, with a focus on English and Portuguese. The authors built a bilingual test set of 170 prompts per language. It combines handpicked items from two public red teaming collections with a smaller in-house set that imitates ordinary users probing the model, for example through euphemisms. Prompts were machine-translated into Portuguese with GPT-4o and then checked by a native speaker. A single annotator labelled every response. Results are reported as a binary safety percentage and as a finer-grained safety score built from +1/−1 categories. Three experiments were run: a five-model comparison, tests with guardrailing and content-moderation system prompts, and a study of how token limits and temperature affect TowerLLM 3.0 models. TowerLLM variants outperformed GPT-4o on safety. Portuguese results were consistently weaker than English, and LLM-based content moderation performed poorly. The work argues for multilingual safety testing and suggests red teaming as a new professional role for translators.

Key points

  • Data: 72 requests come from Haize Labs' Redteaming Resistance Leaderboard dataset, 50 from Babelscape's ALERT, and 48 were newly written, giving 340 requests in total across EN and PT. Twenty of them are two-part prompts. The set is released as Unbabel/Multilingual-Red-Teaming on Hugging Face.
  • Model comparison: Five models were tested: GPT-4o, Vesuvius 3.0, Sugarloaf 3.1, Sugarloaf 4.0 and Vesuvius 4.0. Sugarloaf 3.1 was safest overall, while Vesuvius 4.0 led on the Portuguese safety score. Both beat GPT-4o.
  • Guardrailing prompt: Mistral AI's Safe Prompt improved safety, with the clearest gains in Portuguese.
  • Content moderation: Mistral's Self-reflection Prompt and a Llama Guard 3-8B prompt both gave substandard harm classification. GPT-4o did best on this task, and Vesuvius 3.0 surprisingly outperformed the newer Sugarloaf 3.1.
  • Decoding settings: TowerLLM 3.0 models (Anthill, Sugarloaf, Vesuvius) were tested at 512/1000/2048 tokens with temperature 0 or 1. The 1000-token, temperature-1 setting was safest, while 512 tokens was markedly poor. Higher temperature reduced harmful content but caused more failed requests.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Language models & pretraining · Multilingual & cross-lingual NLP · Bias, fairness & safety

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{pandeiro2026multilingual,
  title = {{A Multilingual Red Teaming–Driven Safety Analysis of LLMs}},
  author = {Pandeiro, Patrícia and Cabarrão, Vera and Moniz, Helena},
  booktitle = {Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1)},
  year = {2026},
  url = {https://aclanthology.org/2026.eamt-1.46/}
}

References

15 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alves, Duarte M., José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza & André F. T. Martins. (2024). Tower: An open Multilingual Large Language Model for Translation-Related Tasks.… find our card
  2. Angwin, J., Jeff Larson, Surya Mattu, & Lauren Kirchner. (2016). Machine Bias. ProPublica. Retrieved March 31, 2025, from https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing ↗
  3. Ayyamperumal, S. G., & Limin Ge. (2024). Current state of LLM Risks and AI Guardrails. arXiv. https://doi.org/10.48550/arXiv.2406.12934 — [doi:10.48550/arxiv.2406.12934](https://doi.org/10.48550/arxiv.2406.12934) ↗
  4. Babelscape. (2024). ALERT: A Comprehensive Benchmark for Assessing Large Language Models’ Safety through Red Teaming. GitHub. Retrieved November 5, 2024, from https://github.com/Babelscape/ALERT ↗
  5. Content moderation with self-reflection. (n.d.). Mistral AI. Retrieved May 13, 2025, from https://docs.mistral.ai/capabilities/guardrailing/\#conten t-moderation-with-self-reflection European Union. (2024). Regulation - EU - 2024/1689 -EN - EUR-Lex. EUR-Lex. Retrieved January 28, 2026, from https://e… ↗
  6. Haize Labs. (2024). Redteaming Resistance Benchmark. GitHub. Retrieved November 5, 2024, from https://github.com/haizelabs/redteaming-resistance-benchmark ↗
  7. Heikkilä, M. (2022). Dutch scandal serves as a warning for Europe over risks of using algorithms. POLITICO. Retrieved March 31, 2025, from https://www.politico.eu/article/dutch-scandal-serves-as-a-warning-for-europe-over-risks-of-using-algorithms/ ↗
  8. Hello GPT-4o. (2024). OpenAI. Retrieved March 5, 2026, from https://openai.com/index/hello-gpt-4o/ ↗
  9. Moniz, H., & Carla Parra Escartín. (Eds.). (2023). Towards Responsible Machine Translation: Ethical and Legal Considerations in Machine Translation (Vol. 4). Springer International Publishing. https://doi.org/10.1007/978-3-031-14689-3 — [doi:10.1007/978-3-031-14689-3](https://doi.org/10.1007/978-3-031-14689-3) ↗
  10. Moorkens, J., Andy Way, & Séamus Lankford. (2024). Automating Translation (1st ed.). Routledge. https://doi.org/10.4324/9781003381280 — [doi:10.4324/9781003381280](https://doi.org/10.4324/9781003381280) ↗
  11. Meta Llama. (n.d.). Llama Guard 3-8B Model Card. GitHub. Retrieved February 15, 2025, from https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard3/8B/MODEL_CARD.md ↗
  12. Obermeyer, Z., Brian Powers, Christine Vogeli, & Sendhil Mullainathan. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342 — [doi:10.1126/science.aax2342](https://doi.org/10.1126/science.aax2342) ↗
  13. System prompt to enforce guardrails. (n.d.). Mistral AI. Retrieved May 13, 2025, from https://docs.mistral.ai/capabilities/guardrailing/\#system -prompt-to-enforce-guardrails Rei, Ricardo, Nuno M. Guerreiro, José Pombal, João Alves, Pedro Teixeirinha, Amin Farajian, & André F. T. Martins. (2025). Tow… ↗
  14. The White House. (2023). Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. The White House. Retrieved August 14, 2025, from https://bidenwhitehouse.archives.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-tru… ↗
  15. Zenko, M. (2015). Red team: How to succeed by thinking like the enemy. Basic Books. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pandeiro-2026-multilingual-red-teaming-driven-safety/