Elamur Research⌕
☰
home / papers / 2023 / Large Language Models are Zero-Shot Rankers for Recommender Systems

Large Language Models are Zero-Shot Rankers for Recommender Systems

Paper · 2023 · Yupeng Hou, Junjie Zhang, Zihan Lin et al.

Summary

The authors ask how well a general-purpose LLM can rank recommendation candidates without task-specific training, and what helps or hurts that ability. They treat recommendation as conditional ranking. A user's chronologically ordered history serves as the condition, and a small pool of items from candidate generators is the set to reorder. Both parts are written into a natural-language instruction for ChatGPT (gpt-3.5-turbo), and the answer is mapped back to item titles by string matching. Experiments on MovieLens-1M and the Amazon Games category use only item titles as text.

The study finds three weaknesses. The LLM largely ignores the order of past interactions. It favours popular items. Its rankings shift depending on where a candidate appears in the prompt. Two fixes help: prompts that highlight the most recent item or add self-derived in-context examples, and bootstrapping, which reranks shuffled candidate lists several times. With these, the zero-shot ranker beats earlier zero-shot recommenders. On candidates pooled from several generators, it also outperforms trained baselines such as Pop and BPRMF. This makes LLMs credible rankers for settings where little in-domain training data exists.

Key points

  • Datasets:
    • ML-1M: 6,040 users, 3,706 items, about 1M interactions.
    • Games: 50,547 users, 16,859 items, 389,718 interactions.
    • Any user or item with under five interactions was filtered out.
  • Evaluation setup:
    • Leave-one-out protocol scored with NDCG@K, implemented in RecBole.
    • Histories capped at 50 items; usually 20 candidates (m = 20).
    • Temperature 0.2; 6,000 sampled users for Games.
  • Three history encodings were tested:
    • Plain sequential prompting.
    • Recency-focused prompting.
    • In-context learning built from prefixes of the user's own sequence.
  • Best configuration: recency-focused prompting with 5 history items plus 3 bootstrapping rounds. On 21-item multi-generator pools, it was second-best on most metrics.
  • Practical notes:
    • GPT-3.5 named items outside the candidate set in roughly 3% of cases.
    • Open models such as LLaMA-2-70B-Chat clearly lagged behind ChatGPT.
    • Code and processed data are available at RUCAIBox/LLMRank.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢TencentShenzhen · China163 works · 21,032 citations🎓Renmin University of ChinaBeijing · China139 works · 17,752 citations🎓University of California San DiegoSan Diego · United States89 works · 54,032 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley and Wayne Xin Zhao. 2023. Large Language Models are Zero-Shot Rankers for Recommender Systems. arXiv:2305.08845.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.08845 · arXiv 2305.08845

@misc{hou2023large,
  title = {{Large Language Models are Zero-Shot Rankers for Recommender Systems}},
  author = {Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley and Wayne Xin Zhao},
  year = {2023},
  doi = {10.48550/arxiv.2305.08845},
  eprint = {2305.08845},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.08845}
}

References

16 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Bao, K., Zhang, J., Zhang, Y., Wang, W., Feng, F., He, X.: Tallrec: An effective and efficient tuning framework to align large language model with recommendation. arXiv preprint arXiv:2305.00447 (2023) 2. Barkan, O., Koenigstein, N.: ITEM2VEC: neural item embedding for collaborative filtering. In…
  2. IEEE (2016). https://doi.org/10.1109/MLSP.2016.7738886, https://doi.org/10.1109/ MLSP.2016.7738886 3. Bonab, H.R., Aliannejadi, M., Vardasbi, A., Kanoulas, E., Allan, J.: Cross-market product recommendation. In: Demartini, G., Zuccon, G., Culpepper, J.S., Huang, Z., Tong, H. (eds.) CIKM. pp. 110–119… — [doi:10.1109/mlsp.2016.7738886](https://doi.org/10.1109/mlsp.2016.7738886) ↗
  3. ACM (2021). https://doi.org/10.1145/3459637.3482493, https://doi.org/10.1145/3459637. 3482493 4. Cao, D., He, X., Nie, L., Wei, X., Hu, X., Wu, S., Chua, T.: Cross-platform app recommendation by jointly modeling ratings and texts. ACM Trans. Inf. Syst. 35(4), 37:1–37:27 (2017). https://doi.org/10.11… — [doi:10.1145/3459637.3482493](https://doi.org/10.1145/3459637.3482493) ↗
  4. IEEE (2021). https://doi.org/10.1109/ICDM51629.2021.00015, https://doi.org/ 10.1109/ICDM51629.2021.00015 9. Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., et al.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt qualit… — [doi:10.1109/icdm51629.2021.00015](https://doi.org/10.1109/icdm51629.2021.00015) ↗
  5. CEUR-WS.org (2020), https://ceur-ws.org/Vol-2715/paper6.pdf 12. Cui, Z., Ma, J., Zhou, C., Zhou, J., Yang, H.: M6-rec: Generative pretrained language models are open-ended recommender systems. arXiv preprint arXiv:2205.08084 (2022) 13. Dai, S., Shao, N., Zhao, H., Yu, W., Si, Z., Xu, C., Sun, Z., Zh… — [arXiv:2205.08084](https://arxiv.org/abs/2205.08084) ↗
  6. Large Language Models are Zero-Shot Rankers for Recommender Systems 14. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL (2019) 15. Ding, H., Ma, Y., Deoras, A., Wang, Y., Wang, H.: Zero-shot recommender syst…
  7. ACM (2018). https://doi.org/10.1145/3219819.3219885, https://doi.org/10. 1145/3219819.3219885 23. Guo, Q., Zhuang, F., Qin, C., Zhu, H., Xie, X., Xiong, H., He, Q.: A survey on knowledge graph-based recommender systems. TKDE 34(8), 3549–3568 (2020) 24. Harper, F.M., Konstan, J.A.: The movielens data… — [doi:10.1145/3219819.3219885](https://doi.org/10.1145/3219819.3219885) ↗
  8. Hou et al. 32. Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.: Mistral 7b. arXiv preprint arXiv:2310.06825 (2023) 33. Kang, W., McAuley, J.: Self-attentive sequential recommendation. In: ICDM (2018) 34… find
  9. Large Language Models are Zero-Shot Rankers for Recommender Systems 51. Roitero, K., Carterette, B., Mehrotra, R., Lalmas, M.: Leveraging behavioral heterogeneity across markets for cross-market training of recommender systems. In: Seghrouchni, A.E.F., Sukthankar, G., Liu, T., van Steen, M. (eds.) W… — [doi:10.1145/3366424.3384362](https://doi.org/10.1145/3366424.3384362) find
  10. ACM (2018). https://doi.org/10.1145/3159652.3159656, https://doi.org/10.1145/3159652.3159656 55. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., Hashimoto, T.B.: Stanford alpaca: An instruction-following llama model (2023) 56. Touvron, H., Lavril, T., Izacard, G., M… ↗
  11. Hou et al. 66. Wu, L., Zheng, Z., Qiu, Z., Wang, H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., Liu, Q., et al.: A survey on large language models for recommendation. arXiv preprint arXiv:2305.19860 (2023) 67. Wu, L., Zheng, Z., Qiu, Z., Wang, H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., Liu,… — [arXiv:2305.19860](https://arxiv.org/abs/2305.19860) find
  12. ACM (2022). https://doi.org/10.1145/3534678.3539120, https://doi. org/10.1145/3534678.3539120 69. Yuan, F., He, X., Karatzoglou, A., Zhang, L.: Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. In: Huang, J.X., Chang, Y., Cheng, X., Kamps, J., Murdock, V.,… ↗
  13. ACM (2020). https://doi.org/10.1145/3397271.3401169, https://doi.org/10.1145/3397271.3401169 77. Zhao, W.X., Lin, Z., Feng, Z., Wang, P., Wen, J.R.: A revisiting study of appropriate offline evaluation for top-n recommendation algorithms. ACM Transactions on Information Systems 41(2), 1–41 (2022) 7… ↗
  14. Large Language Models are Zero-Shot Rankers for Recommender Systems 81. Zheng, B., Hou, Y., Lu, H., Chen, Y., Zhao, W.X., Wen, J.R.: Adapting large language models by integrating collaborative semantics for recommendation. arXiv preprint arXiv:2311.09049 (2023) 82. Zhou, K., Wang, H., Zhao, W.X., Zh… — [arXiv:2311.09049](https://arxiv.org/abs/2311.09049) find
  15. ACM (2019). https://doi.org/10.1145/3357384.3357992, https://doi.org/ 10.1145/3357384.3357992 84. Zhu, F., Wang, Y., Chen, C., Liu, G., Zheng, X.: A graphical and attentional framework for dual-target cross-domain recommendation. In: Bessiere, C. (ed.) IJCAI. pp. 3001–3008. ijcai.org (2020). https:/… — [doi:10.1145/3357384.3357992](https://doi.org/10.1145/3357384.3357992) ↗
  16. ACM (2022). https://doi.org/10.1145/3488560.3498392, https://doi.org/ 10.1145/3488560.3498392 — [doi:10.1145/3488560.3498392](https://doi.org/10.1145/3488560.3498392) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hou-2023-large-language-models-zero-shot/