Elamur Research⌕
☰
home / papers / 2026 / Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge

Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge

Paper · 2026 · Xin Sun, Di Wu, Sijing Qin et al.

Summary

The paper asks whether LLMs used as automatic evaluators (LLM-as-a-Judge) are swayed by disclosed authorship, and whether this resembles how people judge health information. The authors took 150 health question-answer pairs from Yadav et al. (2022). They paired expert-written answers with GPT-4o answers matched in length and format, and crossed true source with a displayed label ("Human Authored" vs. "AI-Generated") in a 2 × 2 counterfactual design. Both lab participants, tracked with an eye tracker, and LLM judges rated trust on a 5-point scale adapted from a validated online-health-trust questionnaire. Both groups trusted identical text more when it was labelled human-written. On the model side, attention concentrated more on the label than on the content, especially under the Human label. This mirrored human gaze. Logit-based uncertainty was higher under AI labels. The authors conclude that provenance labels act as a shared heuristic shortcut. This undermines LLM-as-a-Judge validity and suggests preference alignment may pass human biases to models. They recommend blind or identity-controlled judging.

Key points

  • Data: 150 health QA pairs. Expert answers come from Yadav et al. (2022), and comparable answers were generated with GPT-4o and checked by domain experts.
  • Design: source (human vs. LLM) × label (Human vs. AI), with label swaps that hold content constant. A placebo label condition tests whether the effect comes from label meaning.
  • Human study: eye tracking on predefined AoIs (Label, Content, Rating). Participants looked at the label area more under Human labels and gave Human-labelled text higher trust.
  • LLM internals: models attend more densely to the label region than to the content region, more so under Human labels. Entropy from logits shows greater decision uncertainty under AI labels.
  • Limitations include health-only scope, only two label types, and attention/logit signals that are correlational rather than causal. Materials are released at github.com/XIN-von-SUN/Label-Effects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Social Sciences · 1,954 worksPsychology · 1,649 worksDecision Sciences · 249 worksSocial Psychology · 330 worksPolitical Science and International Relations · 206 worksGeneral Decision Sciences · 4 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Xin Sun, Di Wu, Sijing Qin, Isao Echizen, Abdallah El Ali and Saku Sugawara. 2026. Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1495

@inproceedings{sun2026label,
  title = {{Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge}},
  author = {Xin Sun, Di Wu, Sijing Qin, Isao Echizen, Abdallah {El Ali} and Saku Sugawara},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1495},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1495}
}

References

54 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tobii AB. 2024. Tobii pro lab. Computer software. find
  2. Benjamin R Bates, Sharon Romina, Rukhsana Ahmed, and Danielle Hopson. 2006. The effect of source credibility on consumers’ perceptions of the quality of health information on the internet. Medical informatics and the Internet in medicine, 31(1):45–52. find
  3. Oliver Brady, Paul Nulty, Lili Zhang, Tomás E Ward, and David P McGovern. 2025. Dual-process theory and decision-making in large language models. Nat. Rev. Psychol., 4(12):777–792. find
  4. John T. Cacioppo, Louis G. Tassinary, and Gary G. Berntson. 2016. Strong Inference in Psychophysi-ological Science, page 3–15. Cambridge Handbooks in Psychology. Cambridge University Press. find
  5. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or LLMs as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327, Miami, Florida, USA. Association for Computational Lingu…
  6. Vanessa Cheung, Maximilian Maier, and Falk Lieder. 2025. Large language models show amplified cognitive biases in moral decision-making. Proceedings of the National Academy of Sciences, 122(25):e2412015122. find
  7. Sunhao Dai, Yuqi Zhou, Liang Pang, Weihao Liu, Xiaolin Hu, Yong Liu, Xiao Zhang, Gang Wang, and Jun Xu. 2024. Neural retrievers are biased towards llm-generated content. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, page 526–537, New York, NY, USA.… find
  8. Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünner. 2024. Questioning the survey responses of large language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. Curran Associates Inc. find
  9. Jessica Maria Echterhoff, Yao Liu, Abeer Alessa, Julian McAuley, and Zexue He. 2024. Cognitive bias in decision-making with LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12640–12653, Miami, Florida, USA. Association for Computational Linguistics. find
  10. Abdallah El Ali, Karthikeya Puttur Venkatraj, Sophie Morosoli, Laurens Naudts, Natali Helberger, and Pablo Cesar. 2024. Transparent ai disclosure obligations: Who, what, when, where, why, how. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA ’24, New York, NY… find
  11. Franz Faul, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner. 2007. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods, 39(2):175–191. find
  12. Bertram Gawronski, Dillon M Luke, and Laura A Creighton. 2024. Dual-process theories. In The Oxford Handbook of Social Cognition, Second Edition, pages 319–353. Oxford University Press. find
  13. Ellen R Girden. 1992. ANOVA: Repeated measures. 84. Sage. find
  14. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge. Preprint, arXiv:2411.15594. find
  15. Rajarshi Haldar and Julia Hockenmaier. 2025. Rating roulette: Self-inconsistency in LLM-as-a-judge frameworks. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 24986– 25004, Suzhou, China. Association for Computational Linguistics. find
  16. James W Hardin and Joseph M Hilbe. 2012. Generalized estimating equations, second edition, 2 edition. Chapman & Hall/CRC, Philadelphia, PA. find
  17. Winston Haynes. 2013. Benjamini–Hochberg Method, pages 78–78. Springer New York, New York, NY. find
  18. Maurice Jakesch, Megan French, Xiao Ma, Jeffrey T. Hancock, and Mor Naaman. 2019. Ai-mediated communication: How the perception that profile text was written by ai affects trustworthiness. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, page 1–13, New York,… find
  19. Ruili Jiang, Kehai Chen, Xuefeng Bai, Zhixuan He, Juntao Li, Muyun Yang, Tiejun Zhao, Liqiang Nie, and Min Zhang. 2024. A survey on human preference learning for large language models. arXiv preprint arXiv:2406.11191. — [arXiv:2406.11191](https://arxiv.org/abs/2406.11191) find
  20. Frances C. Johnson, Jennifer E. Rowley, and Laura Sbaffi. 2015. Modelling trust formation in health information contexts. Journal of Information Science, 41:415 – 429. find
  21. Marcel A Just and Patricia A Carpenter. 1980. A theory of reading: From eye fixations to comprehension. Psychol. Rev., 87(4):329–354. find
  22. Tatsuki Kuribayashi, Yohei Oseki, Souhaib Ben Taieb, Kentaro Inui, and Timothy Baldwin. 2025. Large language models are human-like internally. Transactions of the Association for Computational Linguistics, 13:1743–1766.
  23. Walter Laurito, Benjamin Davis, Peli Grietzer, Tomáš Gavenčiak, Ada Böhm, and Jan Kulveit. 2025. Ai–ai bias: Large language models favor communications generated by large language models. Proceedings of the National Academy of Sciences, 122(31):e2415697122. find
  24. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2025a. From generation to judgment: Opportunities and challenges of LLM-as-a-judge. In Proceedings of the 2025 Conference… find
  25. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579.
  26. Qingquan Li, Shaoyu Dou, Kailai Shao, Chao Chen, and Haixiang Hu. 2025b. Evaluating scoring bias in llm-as-a-judge. Preprint, arXiv:2506.22316. — [arXiv:2506.22316](https://arxiv.org/abs/2506.22316) find
  27. Songze Li, Chuokun Xu, Jiaying Wang, Xueluan Gong, Chen Chen, Jirui Zhang, Jun Wang, Kwok-Yan Lam, and Shouling Ji. 2025c. Llms cannot reliably judge (yet?): A comprehensive assessment on the robustness of llm-as-a-judge. Preprint, arXiv:2506.09443. — [arXiv:2506.09443](https://arxiv.org/abs/2506.09443) find
  28. Q.Vera Liao and S. Shyam Sundar. 2022. Designing for responsible trust in ai systems: A communication perspective. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, page 1257–1268, New York, NY, USA. Association for Computing Machinery. find
  29. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. Preprint, arXiv:2303.16634.
  30. Joao Marecos, Duarte Tude Graça, Francisco Goiana-da Silva, Hutan Ashrafian, and Ara Darzi. 2024. Source credibility labels and other nudging interventions in the context of online health misinformation: A systematic literature review. Journalism and Media, 5(2):702–717. find
  31. Arash Marioriyad, Mohammad Hossein Rohban, and Mahdieh Soleymani Baghshah. 2025. The silent judge: Unacknowledged shortcut bias in llm-as-a-judge. Preprint, arXiv:2509.26072. — [arXiv:2509.26072](https://arxiv.org/abs/2509.26072) find
  32. OpenAI. 2024. Gpt-4 technical report. arXiv:2303.08774.
  33. Preprint, Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773…
  34. Jonathan Peirce, Jeremy R Gray, Sol Simpson, Michael MacAskill, Richard Höchenberger, Hiroyuki Sogo, Erik Kastman, and Jonas Kristoffer Lindeløv. 2019. PsychoPy2: Experiments in behavior made easy. Behavior Research Methods, 51(1):195–203. find
  35. Moritz Reis, Florian Reis, and Wilfried Kunde. 2024. Influence of believed AI involvement on the perception of digital medical advice. Nature Medicine. find
  36. Bernard Rosner, Robert J Glynn, and Mei-Ling T Lee. 2006. The wilcoxon signed rank test for paired comparisons of clustered data. Biometrics, 62(1):185– 192. find
  37. Jennifer E. Rowley, Frances C. Johnson, and Laura Sbaffi. 2015. Students’ trust judgements in online health information seeking. Health Informatics Journal, 21:316 – 327. find
  38. Nicolas Scharowski, Michaela Benk, Swen J. KÃŒhne, Léane Wettstein, and Florian BrÃŒhlmann. 2023. Certification Labels for Trustworthy AI: Insights From an Empirical Mixed-Method Study. In 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 248–260, Chicago IL USA. ACM. find
  39. Kayla Schroeder and Zach Wood-Doughty. 2025. Can you trust llm judgments? reliability of llm-as-a-judge. Preprint, arXiv:2412.12509. — [arXiv:2412.12509](https://arxiv.org/abs/2412.12509) find
  40. Huanxin Sheng, Xinyi Liu, Hangfeng He, Jieyu Zhao, and Jian Kang. 2025. Analyzing uncertainty of LLM-as-a-judge: Interval evaluations with conformal prediction. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 11297–11339, Suzhou, China. Association fo… find
  41. Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. 2025. Judging the judges: A systematic study of position bias in llm-as-a-judge. Preprint, arXiv:2406.07791.
  42. Evangelia Spiliopoulou, Riccardo Fogliato, Hanna Burnsky, Tamer Soliman, Jie Ma, Graham Horwood, and Miguel Ballesteros. 2025. Play favorites: A statistical method to measure self-bias in llm-as-a-judge. Preprint, arXiv:2508.06709. — [arXiv:2508.06709](https://arxiv.org/abs/2508.06709) find
  43. Xin Sun, Rongjun Ma, Shu Wei, Pablo Cesar, Jos A. Bosch, and Abdallah El Ali. 2025. Understanding trust toward human versus ai-generated health information through behavioral and physiological sensing. International Journal of Human-Computer Studies, page 103714. find
  44. Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. 2024. Do LLMs exhibit human-like response biases? a case study in survey design. Transactions of the Association for Computational Linguistics, 12:1011–1026.
  45. Qian Wang, Zhanzhi Lou, Zhenheng Tang, Nuo Chen, Xuandong Zhao, Wenxuan Zhang, Dawn Song, and Bingsheng He. 2025a. Assessing judging bias in large reasoning models: An empirical study. Preprint, arXiv:2504.09946. — [arXiv:2504.09946](https://arxiv.org/abs/2504.09946) find
  46. Yidong Wang, Yunze Song, Tingyuan Zhu, Xuanwang Zhang, Zhuohao Yu, Hao Chen, Chiyu Song, Qiufeng Wang, Cunxiang Wang, Zhen Wu, Xinyu Dai, Yue Zhang, Wei Ye, and Shikun Zhang. 2025b. Trustjudge: Inconsistencies of llm-as-a-judge and how to alleviate them. Preprint, arXiv:2509.21117. — [arXiv:2509.21117](https://arxiv.org/abs/2509.21117) find
  47. Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2025. Self-preference bias in llm-as-a-judge. Preprint, arXiv:2410.21819.
  48. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11–20, Hong Kong, China. Association for…
  49. Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye, Shiyang Lai, Kai Shu, Jindong Gu, Adel Bibi, Ziniu Hu, David Jurgens, James Evans, Philip H.S. Torr, Bernard Ghanem, and Guohao Li. 2024. Can large language model agents simulate human trust behavior? In Proceedings of the 38th International Conference… find
  50. Shweta Yadav, Deepak Gupta, and Dina Demner-Fushman. 2022. Chq-summ: A dataset for consumer healthcare question summarization. Preprint, arXiv:2206.06581. — [arXiv:2206.06581](https://arxiv.org/abs/2206.06581) find
  51. Haoyan Yang, Runxue Bao, Cao Xiao, Jun Ma, Parminder Bhatia, Shangqian Gao, and Taha Kass-Hout. 2025. Any large language model can be a reliable judge: Debiasing with a reasoning-based bias detector. Preprint, arXiv:2505.17100. — [arXiv:2505.17100](https://arxiv.org/abs/2505.17100) find
  52. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. Preprint, arXiv:2410.02736.
  53. Yidan Yin, Nan Jia, and Cheryl J. Wakslak. 2024. Ai can help people feel heard, but an ai label diminishes this impact. Proceedings of the National Academy of Sciences, 121(14):e2319112121. find
  54. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-label-effects-shared-heuristic-reliance/