Elamur Research⌕
☰
home / papers / 2026 / Whom to Trust? Analyzing the Divergence Between User Satisfaction and LLM-as-a-Judge in E-Commerce RAG Systems

Whom to Trust? Analyzing the Divergence Between User Satisfaction and LLM-as-a-Judge in E-Commerce RAG Systems

Paper · 2026 · Arif Türkmen and Kaan Efe Keleş

Summary

Production RAG systems need scalable evaluation because A/B tests are slow and expensive. It is unclear, however, whether LLM judges reflect what real customers think. The authors study this on the Trendyol QA Assistant, a Turkish e-commerce service. It answers product questions by retrieving similar historical Q/A pairs with TY-ecomm-embed-multilingual-base-v1.2.0 and then conditioning Trendyol-LLM-8B-T1 on them. Using roughly 150k live interactions with thumbs-up/down labels and optional dissatisfaction reasons, they compare user verdicts against four few-shot LLM judges. They also add factor-level diagnostics: question topic, intent, context relevance, persona consistency and answer length. The judges approximate the overall satisfaction rate, but their agreement with individual users is only slight to fair. Agreement drops further when the retrieved context is weak or the question concerns a specific seller. Judges also blame failures on factual errors far more often than users do, while users mostly complain about incomplete answers. The study shows that relying on a single judge can mislead production teams. It argues for combining judge ensembles with direct user feedback.

Key points

  • Data and judges: The study uses about 150k Turkish production interactions with a user satisfaction rate of 77.2%. The judges are GPT-4o, o4-mini, Gemini 2.5 Pro and Claude 4 Sonnet. They received a 10-example few-shot prompt that included the query, retrieved pairs, the answer and the system prompts.
  • Agreement with users: Exact-match agreement ranged from 64.5% (GPT-4o, κ 0.15) to 75.0% (Claude 4 Sonnet, κ 0.27). The reasoning model o4-mini clearly outperformed GPT-4o (72.5%, κ 0.23).
  • Low-relevance context: Human satisfaction falls to 61.1% on low-relevance context slices, and judge κ falls to 0.06–0.16. Judges, which can see the retrieved evidence, penalize these answers more harshly than users do.
  • Dissatisfaction reasons: Users cite "insufficient/incomplete" in 62.3% of negative cases and "misleading/incorrect" in 14.2%. Judges assign "misleading/incorrect" in 39–62% of their negative verdicts and almost never choose "unclear."
  • Seller-related queries: Performance depends on the query segment. For seller-specific questions, Claude 4 Sonnet's κ drops from 0.27 to 0.15, while GPT-4o's rises slightly from 0.15 to 0.19. This motivates segment-aware, multi-judge evaluation combined with human feedback.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Arif Türkmen and Kaan Efe Keleş. 2026. Whom to Trust? Analyzing the Divergence Between User Satisfaction and LLM-as-a-Judge in E-Commerce RAG Systems. Proceedings of the First Workshop on Multilingual Multicultural Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.mme-main.12

@inproceedings{turkmen2026whom,
  title = {{Whom to Trust? Analyzing the Divergence Between User Satisfaction and LLM-as-a-Judge in E-Commerce RAG Systems}},
  author = {Arif Türkmen and Kaan Efe Keleş},
  booktitle = {Proceedings of the First Workshop on Multilingual Multicultural Evaluation},
  year = {2026},
  doi = {10.18653/v1/2026.mme-main.12},
  url = {https://doi.org/10.18653/v1/2026.mme-main.12}
}

References

6 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  2. Robert Friel, Masha Belyi, and Atindriyo Sanyal. 2025. Ragbench: Explainable benchmark for retrieval-augmented generation systems. Preprint, arXiv:2407.11005. — [arXiv:2407.11005](https://arxiv.org/abs/2407.11005) find
  3. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge. Preprint, arXiv:2411.15594. find
  4. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-augmented generation for knowledge-intensive nlp tasks. Preprint, arXiv:2005.11401.
  5. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. Preprint, arXiv:2410.02736.
  6. Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, and Zhaofeng Liu. 2025. Evaluation of Retrieval-Augmented Generation: A Survey, page 102–120. Springer Nature Singapore. find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/turkmen-2026-whom-trust-analyzing-divergence-between/