Elamur Research⌕
☰
home / papers / 2026 / Multimodal Item Scoring for Natural Language Recommendation via Gaussian Process Regression with LLM Relevance Judgments

Multimodal Item Scoring for Natural Language Recommendation via Gaussian Process Regression with LLM Relevance Judgments

Paper · 2026 · Yifan Simon Liu, Qianfeng Wen, Jiazhou Liang et al.

Summary

Natural Language Recommendation (NLRec) ranks items, each described by several text passages, against a free-form user request. Standard Dense Retrieval (DR) scores passages by embedding similarity to the request alone. This assumes relevance peaks in one region around the query. In practice, a request such as a family vacation city can touch several distinct themes, and the relevant passages sit far apart in embedding space.

The authors propose GPR-LLM. For each query, it selects a small subset of passages using an ε-greedy mix of top DR hits and random exploration. An LLM grades these passages pointwise. A Gaussian Process Regression model is then fitted to the graded passages, with the query embedding included as a maximum-relevance anchor. The GPR posterior mean scores every passage, and the passage scores are aggregated into item scores.

With a linear kernel and only the query as signal, the method reduces exactly to DR. Localized kernels such as RBF let each judged anchor shape its own neighbourhood, so the model captures several relevance modes. Across four datasets and three LLM backbones, GPR-LLM beats the baselines by up to 65%. It also stays competitive when it receives far fewer LLM labels.

Key points

  • Anchor set and kernels: the GPR training set combines the query embedding (labelled with the maximum score) and R ≪ N LLM-judged passages. The authors test three kernels: dot product, cosine, and RBF.
  • Kernel comparison: RBF consistently outperforms linear kernels, which the authors credit to its stronger locality.
  • Baselines: BM25, DR, a cross-encoder, and pointwise LLM relevance scoring, all under the same LLM labeling budget. The gains reach up to 65% across 4 NLRec datasets and 3 LLM backbones.
  • Item scoring: each item's score is the mean or max over its top-T passage scores.
  • Visual evidence: t-SNE plots show that DR's top-100 passages cluster tightly around the query, while GPR-LLM's top-100 spread across multiple regions.
  • Stated limitations: results depend on the quality of the LLM judgments. The ε-greedy sampler relies on uniform random exploration rather than active learning, and the aggregation function is fixed. The work comes from the University of Toronto, Sungkyunkwan University, and the Vector Institute.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TorontoToronto · Canada96 works · 142,007 citations🔬Vector InstituteToronto · Canada33 works · 4,535 citations🎓Sungkyunkwan UniversitySuwon · South Korea9 works · 195 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Yifan Simon Liu, Qianfeng Wen, Jiazhou Liang, Mark Zhao, Justin Cui, Anton Korikov, Armin Toroghi, Junyoung Kim and Scott Sanner. 2026. Multimodal Item Scoring for Natural Language Recommendation via Gaussian Process Regression with LLM Relevance Judgments. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1836

@inproceedings{liu2026multimodal,
  title = {{Multimodal Item Scoring for Natural Language Recommendation via Gaussian Process Regression with LLM Relevance Judgments}},
  author = {Yifan Simon Liu, Qianfeng Wen, Jiazhou Liang, Mark Zhao, Justin Cui, Anton Korikov, Armin Toroghi, Junyoung Kim and Scott Sanner},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1836},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1836}
}

References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jafar Afzali, Aleksander Mark Drzewiecki, and Krisztian Balog. 2021. POINTREC: A test collection for narrative-driven point-of-interest recommendation. In Proceedings of the 44th International ACM SI-GIR Conference on Research and Development in Information Retrieval, pages 2478–2484. find
  2. Toine Bogers, Maria Gäde, Marijn Koolen, Vivien Petras, and Mette Skov. 2018. What was this movie about this chick?: A comparative study of relevance aspects in book and movie discovery. In Transforming Digital Worlds, volume 10766 of Lecture Notes in Computer Science, pages 323–334, Cham. Springer. find
  3. Toine Bogers, Maria Gäde, Marijn Koolen, Vivien Petras, and Mette Skov. 2019. Looking for an amazing game i can relax and sink hours into...: A study of relevance aspects in video game discovery. In Information in Contemporary Society, volume 11420 of Lecture Notes in Computer Science, pages 503–515… find
  4. Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages…
  5. Difan Jiao, Qianfeng Wen, Blair Yang, Zhenwei Tang, and Ashton Anderson. 2026. Thinktwice: Jointly optimizing large language models for reasoning and self-refinement. arXiv preprint arXiv:2604.01591. — [arXiv:2604.01591](https://arxiv.org/abs/2604.01591) find
  6. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli…
  7. Jiazhou Liang, Yifan Simon Liu, David Guo, Minqi Sun, Yilun Jiang, and Scott Sanner. 2026. Evaluating scene-based in-situ item labeling for immersive conversational recommendation. Preprint, arXiv:2604.09698. find
  8. Yifan Liu, Gelila Tilahun, Xinxiang Gao, Qianfeng Wen, and Michael Gervers. 2025a. A comparative study of static and contextual embeddings for analyzing semantic changes in medieval Latin charters. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 182–192, Abu… find
  9. Yifan Liu, Qianfeng Wen, Mark Zhao, Jiazhou Liang, and Scott Sanner. 2025b. MA-DPR: Manifold-aware distance metrics for dense passage retrieval. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31085– 31103, Suzhou, China. Association for Computational… find
  10. Yifan Simon Liu, Ruifan Wu, Liam Gallagher, Jiazhou Liang, Armin Toroghi, and Scott Sanner. 2026. Semantic xpath: Structured agentic memory access for conversational ai. arXiv preprint arXiv:2603.01160.
  11. Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero-shot listwise document reranking with a large language model. preprint arXiv:2305.02156. arXiv Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with BERT. arXiv preprint arXiv:1910.14424. find
  12. OpenAI. 2024. GPT-4o system card. Technical report, OpenAI.
  13. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2024. Large language models are effective text rankers with pairwise ranking prompting. In Findings of the Association for Computational Lingu…
  14. Carl E. Rasmussen and Christopher K. I. Williams. 2006. Gaussian Processes for Machine Learning. MIT Press. find
  15. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  16. Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at TREC-3. In Proceedings of The Third Text REtrieval Conference (TREC-3), pages 109–126. National Institute of Standards and Technology.
  17. Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. Improving passage retrieval with zero-shot question generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3781–3797, Abu Dhabi,…
  18. Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Yibin Lei, Tianyi Zhou, Michael Blumenstein, and Daxin Jiang. 2024. Retrieval-augmented retrieval: Large language models are strong zero-shot retriever. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15933–15946, Bangk…
  19. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1491…
  20. Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. 2024. UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor. arXiv preprint arXiv:2406.06519. — [arXiv:2406.06519](https://arxiv.org/abs/2406.06519) find
  21. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579–2605. find
  22. Qianfeng Wen, Yifan Liu, Justin Cui, Joshua Zhang, Anton Korikov, George-Kirollos Saad, and Scott Sanner. 2025a. A simple but effective elaborative query reformulation approach for natural language recommendation. arXiv preprint arXiv:2510.02656. — [arXiv:2510.02656](https://arxiv.org/abs/2510.02656) find
  23. Qianfeng Wen, Yifan Liu, Joshua Zhang, George Saad, Anton Korikov, Yury Sambale, and Scott Sanner. 2024. Elaborative subtopic query reformulation for broad and indirect queries in travel destination recommendation. In Proceedings of the 1st Workshop on Risks, Opportunities, and Evaluation of Generat… — [arXiv:2410.01598](https://arxiv.org/abs/2410.01598) find
  24. Qianfeng Wen, Zhenwei Tang, and Ashton Anderson. 2025b. Chessqa: Evaluating large language models for chess understanding. arXiv preprint arXiv:2510.23948. — [arXiv:2510.23948](https://arxiv.org/abs/2510.23948) find
  25. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, and 1 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods i…
  26. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  27. Haochen Zhang, Anton Korikov, Parsa Farinneya, Mohammad Mahdi Abdollah Pour, Manasa Bharadwaj, Ali Pesaranghader, Xi Yu Huang, Yi Xin Lok, Zhaoqi Wang, Nathan Jones, and 1 others. 2023. Recipe-MPR: A test collection for evaluating multi-aspect preference-based natural language retrieval. In Proceedi… find
  28. Ciyou Zhu, Richard H. Byrd, Peihuang Lu, and Jorge Nocedal. 1997. Algorithm 778: L-bfgs-b, fortran routines for large scale bound constrained optimization. ACM Transactions on Mathematical Software, 23(4):550–560. find
  29. Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2024. Beyond yes and no: Improving zero-shot LLM rankers via scoring fine-grained relevance labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingu… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-multimodal-item-scoring-natural-language/