Elamur Research⌕
☰
home / papers / 2026 / Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization

Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization

Paper · 2026 · Tiancheng Xing, Jerry Li, Yixuan Du et al.

Summary

LLMs are now widely used to rerank candidate items in retrieval and recommendation systems. This paper asks whether an attacker can push one chosen item to the top of a listwise LLM ranking by adding a short, natural-looking phrase to that item's own description. Earlier attacks tended to trade stealth for strength: they either relied on obvious override instructions or produced awkward text. The authors propose Rank Anything First (RAF), which builds the injected suffix one token at a time. In the first stage, Greedy Coordinate Gradient produces a shortlist of tokens using the gradient of the ranking objective together with a readability signal. In the second stage, each shortlisted token is scored with exact ranking and readability losses, which are balanced by an entropy-based dynamic weight, and the final token is drawn by temperature-controlled sampling. In product-ranking experiments on several open-source LLMs, RAF promoted target items more strongly and more stably than baselines such as StealthRank, and its text stayed fluent. The optimized prompts also transferred across models. The results suggest that LLM rerankers are systematically open to manipulation.

Key points

  • Threat model: an LLM reranker receives a query and a list of products, each with brand, price, and description. The attacker may edit only the target item's description. Evaluation uses randomized input orders.
  • Loss functions:
    • The ranking loss is the cross-entropy toward generating the target product's name.
    • The readability loss is the negative log-likelihood of the candidate token, given the description plus the suffix built so far.
  • Token efficiency: a RAF suffix of only 10 tokens already outperforms competing methods that use 30-token prompts. Ablations show that simply adding extra tokens, with no optimization toward the target objective, fails to raise the item's rank.
  • Failure modes:
    • Fluent but ineffective: when next-token entropy is low, readability dominates and the suffix adds no persuasive push.
    • Mode collapse: different products converge on the same generic phrases, such as "stainless steel body". Duplicate-phrase detection across listings could catch this.
  • Code: available at github.com/glad-lab/RAF. The authors note that their reranking pipeline is simplified and that real-world effectiveness still needs validation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States162 works · 35,836 citations🎓National University of SingaporeSingapore · Singapore180 works · 34,304 citations🎓Georgetown UniversityWashington · United States88 works · 4,732 citations🎓Arizona State UniversityTempe · United States50 works · 2,032 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Tiancheng Xing, Jerry Li, Yixuan Du and Xiyang Hu. 2026. Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.413

@inproceedings{xing2026are,
  title = {{Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization}},
  author = {Tiancheng Xing, Jerry Li, Yixuan Du and Xiyang Hu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.413},
  url = {https://doi.org/10.18653/v1/2026.acl-long.413}
}

References

33 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  2. DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, and 69 others. 2024. Deepseek llm: Scaling open-source language models with longterm… find
  3. Yixuan Du, Chenxiao Yu, Haoyan Xu, Ziyi Wang, Yue Zhao, and Xiyang Hu. 2026. Multimodal generative engine optimization: Rank manipulation for vision-language model rankers. Preprint, arXiv:2601.12263. find
  4. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  5. Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. 2024. Cold-attack: Jailbreaking llms with stealthiness and controllability. Preprint, arXiv:2402.08679. — [arXiv:2402.08679](https://arxiv.org/abs/2402.08679) find
  6. Xiyang Hu. 2025. Dynamics of adversarial attacks on large language model-based search engines. Preprint, arXiv:2501.00745. — [arXiv:2501.00745](https://arxiv.org/abs/2501.00745) find
  7. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  8. Sein Kim, Hongseok Kang, Seungyoon Choi, Donghyun Kim, Minchul Yang, and Chanyoung Park. 2024. Large language models meet collaborative filtering: An efficient all-round llm-based recommender system. Preprint, arXiv:2404.11343. — [arXiv:2404.11343](https://arxiv.org/abs/2404.11343) find
  9. Aounon Kumar and Himabindu Lakkaraju. 2024. Manipulating large language models to increase product visibility. Preprint, arXiv:2404.07981. — [arXiv:2404.07981](https://arxiv.org/abs/2404.07981) find
  10. Jiate Li, Defu Cao, Li Li, Wei Yang, Yuehan Qin, Chenxiao Yu, Tiannuo Yang, Ryan A. Rossi, Yan Liu, Xiyang Hu, and Yue Zhao. 2026. "someone hid it": Query-agnostic black-box attacks on llm-based retrieval. Preprint, arXiv:2602.00364. — [arXiv:2602.00364](https://arxiv.org/abs/2602.00364) find
  11. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, and et al. 2022. Holistic evaluation of language models. Preprint, arXiv:2211.09110. find
  12. Weiran Lin, Anna Gerchanovsky, Omer Akgul, Lujo Bauer, Matt Fredrikson, and Zifan Wang. 2025. Llm whisperer: An inconspicuous attack to bias llm responses. Preprint, arXiv:2406.04755. — [arXiv:2406.04755](https://arxiv.org/abs/2406.04755) find
  13. Qidong Liu, Xiangyu Zhao, Yuhao Wang, Yejing Wang, Zijian Zhang, Yuqi Sun, Xiang Li, Maolin Wang, Pengyue Jia, Chong Chen, Wei Huang, and Feng Tian. 2025. Large language model enhanced recommender systems: A survey. Preprint, arXiv:2412.13432. — [arXiv:2412.13432](https://arxiv.org/abs/2412.13432) find
  14. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024a. Autodan: Generating stealthy jailbreak prompts on aligned large language models. Preprint, arXiv:2310.04451.
  15. Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024b. Automatic and universal prompt injection attacks against large language models. Preprint, arXiv:2403.04957. — [arXiv:2403.04957](https://arxiv.org/abs/2403.04957) find
  16. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2024c. Prompt injection attack against llm-integrated applications. Preprint, arXiv:2306.05499.
  17. Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero-shot listwise document reranking with a large language model. Preprint, arXiv:2305.02156. find
  18. Liang-bo Ning, Shijie Wang, Wenqi Fan, Qing Li, Xin Xu, Hao Chen, and Feiran Huang. 2024. Cheatagent: Attacking llm-empowered recommender systems via llm agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, page 2284–2295. ACM. find
  19. Qiyao Peng, Hongtao Liu, Hua Huang, Yuhan Chen, Lianghao Xia, Chenxu Zhu, Zhenwei Tang, Liang Zhang, Yaochen Zhu, Jianxin Li, and Xiangnan He. 2025. A survey on llm-powered agents for recommender systems. Preprint, arXiv:2502.10050. find
  20. Samuel Pfrommer, Yatong Bai, Tanmay Gautam, and Somayeh Sojoudi. 2024. Ranking manipulation for conversational search engines. Preprint, arXiv:2406.03589. find
  21. Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021. The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models. Preprint, arXiv:2101.05667. — [arXiv:2101.05667](https://arxiv.org/abs/2101.05667) find
  22. Rui Pu, Chaozhuo Li, Rui Ha, Litian Zhang, Lirong Qiu, and Xi Zhang. 2024. BaitAttack: Alleviating intention shift in jailbreak attacks via adaptive bait crafting. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15654–15668, Miami, Florida, USA. Assoc… find
  23. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2024. Large language models are effective text rankers with pairwise ranking prompting. In Findings of the Association for Computational Lingu…
  24. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. Preprint, arXiv:2307.15043.
  25. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. Preprint, arXiv:2308.03825.
  26. Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. 2025. Optimization-based prompt injection attack to llm-as-a-judge. Preprint, arXiv:2403.17710. — [arXiv:2403.17710](https://arxiv.org/abs/2403.17710) find
  27. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large language models as re-ranking agents. arXiv preprint arXiv:2304.09542.
  28. Yiming Tang, Yi Fan, Chenxiao Yu, Tiankai Yang, Yue Zhao, and Xiyang Hu. 2025. Stealthrank: Llm ranking manipulation via stealthy prompt optimization. Preprint, arXiv:2504.05804. — [arXiv:2504.05804](https://arxiv.org/abs/2504.05804) find
  29. Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, Hui Xiong, and Enhong Chen. 2024. A survey on large language models for recommendation. Preprint, arXiv:2305.19860. — [arXiv:2305.19860](https://arxiv.org/abs/2305.19860) find
  30. Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. 2024. Jailbreak attacks and defenses against large language models: A survey. Preprint, arXiv:2407.04295.
  31. Jinghao Zhang, Yuting Liu, Qiang Liu, Shu Wu, Guibing Guo, and Liang Wang. 2024. Stealthy attack on large language model based recommendation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5839–5857, Bangkok, Thailand. Assoc… find
  32. Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2023. Autodan: Interpretable gradient-based adversarial attacks on large language models. Preprint, arXiv:2310.15140. find
  33. Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2024. Beyond yes and no: Improving zero-shot LLM rankers via scoring fine-grained relevance labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingu… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xing-2026-are-llms-reliable-rankers-rank/