LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language
Paper · 2025 · Yubin Ge, Neeraja Kirtane, Hao Peng et al.
Summary
The paper asks whether safety-aligned LLMs can be pushed into producing stereotyped or toxic output when a harmful request is framed in academic terms. The idea draws on Cialdini's authority principle. The authors collected real social-science and psychology papers that discuss possible upsides of stereotyping and had GPT-4o condense them into a one-sided summary. That summary went into the system message, together with an instruction to continue a dialogue in a strongly biased way and to justify it. A second variant has an LLM invent fictitious paper titles and abstracts, which requires no expert knowledge from the attacker. Inputs came from StereoSet and GPT-4-generated opening utterances. Targets included GPT-4o, GPT-4o-mini, GPT-4, Gemini, Cohere and Llama3.1 models. GPT-4 graded bias and the Perspective API measured toxicity. Both real and fabricated "evidence" raised bias and toxicity compared with zero-shot, DAN and role-play baselines. Simple defenses did little to stop the attack. The authors argue this calls into question how scientific text is used to train LLMs.
Key points
- Two attack variants are compared: Sci-Paper Based Persuasion, built on real papers, and fabricated-paper persuasion, built on invented titles and abstracts. Both increase stereotypical bias and toxicity.
- Bias is rated by GPT-4 as an LLM judge on a five-level rubric, running from no bias to extreme bias. Toxicity is measured with the Perspective API.
- Ablations on GPT-4o show that stripping author names or venues from the summaries lowers bias scores. Rewriting the summaries in a simpler, K12-level style slightly raises them. GPT-4o-mini is largely unaffected by these changes.
- In multi-turn dialogues (GPT-4o, GPT-4o-mini, GPT-3.5-turbo), average bias keeps rising with each turn.
- Mutation defenses (Jain et al., 2023) mostly fail:
- On GPT-4o, rephrasing cuts bias from 1.71 to 1.01, while retokenizing only reaches 1.63.
- On GPT-4o-mini, bias goes from 2.59 to 2.74 with rephrasing and 2.65 with retokenizing.
- On Llama3-405B, bias goes from 2.30 to 2.63 with rephrasing and 2.68 with retokenizing.
Institutions
United StatesFields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works
🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works
Cite
Cite as Yubin Ge, Neeraja Kirtane, Hao Peng and Dilek Hakkani-Tur. 2025. LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language. arXiv:2501.14073.
@misc{ge2025llms,
title = {{LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language}},
author = {Yubin Ge, Neeraja Kirtane, Hao Peng and Dilek Hakkani-Tur},
year = {2025},
doi = {10.48550/arxiv.2501.14073},
eprint = {2501.14073},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2501.14073}
}References
- concerns about the reliance on scientific data in the training of LLMs and the potential misuse of such models to reinforce societal stereotypes. find
- Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. \#\# Ethics Statement
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/ge-2025-llms-vulnerable-malicious-prompts-disguised/