Elamur Research⌕
☰
home / papers / 2026 / Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimation

Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimation

Paper · 2026 · Bhaktipriya Radharapu, Eshika Saxena, Kenneth Li et al.

Summary

LLM judges are now widely used to score model outputs and supply reward signals, but they tend to be overconfident. Existing ways of getting confidence from them are either poorly calibrated, as with self-reported confidence, or costly, as with methods that sample many generations. The authors take the judge's residual-stream activation at the final token of a reasoning judge and fit a linear probe on it, trained with a Brier-score loss to predict whether the verdict is correct. A sigmoid turns the probe output into a probability. The judge model itself is never retrained.

Probes are trained on the PPE data, which covers both preference and correctness judgments. They are then tested on JudgeBench and RewardBench as out-of-distribution sets. The evaluation spans dense and MoE judges, both prompted and fine-tuned.

The probes are better calibrated than verbalized, consistency and majority-vote baselines, at roughly a tenth of the compute. They also transfer well to new domains. Their estimates are cautious, though, so they can trail verbalized confidence on easy benchmarks. That caution may be an advantage in risk-sensitive settings where false positives are costly.

Key points

  • Training data: PPE Preference (10.2K human preference pairs in 121 languages) and PPE Correctness (12.7K pairs from MMLU-Pro, MATH, GPQA, MBPP-Plus and IFEval).
  • Out-of-distribution data: JudgeBench (620 samples) and RewardBench (3K samples).
  • Judges tested:
    • Prompted Llama 8B/70B and Qwen 32B.
    • Fine-tuned J1 models.
    • MoE judges Llama-Scout (109B) and GPT-OSS-20B.
  • Main results: Calibration is measured with Kuiper and ECE. Probes beat every baseline on PPE-Correctness for all models and on JudgeBench. On RewardBench they sometimes lag behind verbalized confidence.
  • Ablations: Brier loss outperforms binary cross-entropy and focal loss, and probes on middle layers calibrate best. Probe memory scales with layers × hidden_dim, not with token count.
  • Limitations: The method needs access to the judge's hidden states and labeled ground-truth verdicts. The probe may also need retraining whenever the judge changes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Meta (United Kingdom)London · United Kingdom46 works · 18,033 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksInterpretability · 130 works

Cite

Cite as Bhaktipriya Radharapu, Eshika Saxena, Kenneth Li, Chenxi Whitehouse, Adina Williams and Nicola Cancedda. 2026. Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.14

@inproceedings{radharapu2026calibrating,
  title = {{Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimation}},
  author = {Bhaktipriya Radharapu, Eshika Saxena, Kenneth Li, Chenxi Whitehouse, Adina Williams and Nicola Cancedda},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.14},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.14}
}

References

79 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Meta AI. 2025. Llama-4-scout-17b-16e-instruct. https://huggingface.co/meta-llama/ Llama-4-Scout-17B-16E-Instruct. Llama 4 Scout is a 17B parameter, 16 expert mixture-of-experts multimodal model. Licensed under the Llama 4 Community License Agreement. ↗
  2. AI@Meta. 2024. Llama 3 model card. find
  3. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24.
  4. Imanol Arrieta-Ibarra, Paman Gujral, Jonathan Tannen, Mark Tygert, and Cherie Xu. 2022. Metrics of calibration for probabilistic predictions. Journal of Machine Learning Research, 23(351):1–54. find
  5. Amos Azaria and Tom M Mitchell. 2023. The internal state of an llm knows when it’s lying. Findings of the Association for Computational Linguistics.
  6. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron Jinhe Bi, Danqi Yan, Yifan Wang, Wenke Huang, Haokun Chen, Guancheng Wan, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, and Yunpu Ma. 2025. Cot-kinetics: A… — [arXiv:2505.13408](https://arxiv.org/abs/2505.13408) find
  7. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering latent knowledge in language models without supervision. International Conference on Learning Representations. find
  8. Jiuhai Chen and Jonas Mueller. 2024. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5186–5200, Bangkok, Thailand. Association for…
  9. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176.
  10. Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. 2025. Judgelrm: Large reasoning models as a judge. Preprint, arXiv:2504.00050. — [arXiv:2504.00050](https://arxiv.org/abs/2504.00050) find
  11. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, and 1 others. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machin…
  12. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30. find
  13. Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas. 2025. Beyond binary rewards: Training lms to reason about their uncertainty. arXiv preprint arXiv:2507.16806. find
  14. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  15. Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039–30069.
  16. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv: 2101.03961. find
  17. Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph Gonzalez, and I. Stoica. 2024. How to evaluate reward models for rlhf. International Conference on Learning Representations. find
  18. Leo Gao, John Schulman, and Jacob Hilton. 2022. Scaling laws for reward model overoptimization. arXiv preprint arXiv: 2210.10760.
  19. W. Brier Glenn. 1950. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1–3. find
  20. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. arXiv preprint arXiv: 1706.04599. find
  21. Yuheng Huang, Jiayang Song, Zhijie Wang, Huaming Chen, Likun Shen, Xin Lei, and Felix Juefei-Xu Wang. 2023. Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv preprint arXiv:2307.10236. — [arXiv:2307.10236](https://arxiv.org/abs/2307.10236) find
  22. Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, and Nicola Cancedda. 2025. Calibrating verbal uncertainty as a linear feature to reduce hallucinations. arXiv preprint arXiv: 2503.14477.
  23. Jaehun Jung, Faeze Brahman, and Yejin Choi. 2024. Trust or escalate: Llm judges with provable guarantees for human agreement. arXiv preprint arXiv:2407.18370. find
  24. Seungone Kim, Juyoung Shin, Yejin Kwak, Yujin Zhao, Jungwoo Jang, Ahra Hwang, Geon Kim, Hyeonbin Kim, Junho Yoon, Hahyeon Park, and 1 others. 2024b. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535.
  25. Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew M Bean, Katerina Margatina, Rafael Mosquera-Gomez, Juan Ciro, Max Bartolo, Adina Williams, He He, and 1 others. 2024. The prism alignment dataset: What participatory, representative and individualised human feedback reveals about the subje… find
  26. Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. 2024. Semantic entropy probes: Robust and cheap hallucination detection in llms. arXiv preprint arXiv: 2406.15927. — [arXiv:2406.15927](https://arxiv.org/abs/2406.15927) find
  27. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. International Conference on Learning Representations. find
  28. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv: 2403.13787. find
  29. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2023. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv: 2309.00267. find
  30. Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2024a. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470.
  31. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022.… find
  32. Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. 2024. Calibration-tuning: Teaching large language models to know what they don’t know. In Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024), pages 1–14, St Julians, Malta… find
  33. Seungone Kim, Juyoung Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Sungdong Shin, Seongjin Kim, James Thorne, and 1 others. 2024a. Prometheus: Inducing fine-grained evaluation capability in language models. arXiv preprint arXiv:2310.08491.
  34. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2024b. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36.
  35. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024c. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv: 2406.11939.
  36. Yibo Li, Miao Xiong, Jiaying Wu, and Bryan Hooi. 2025a. Conftuner: Training large language models to express their confidence verbally. arXiv preprint arXiv: 2508.18847. — [arXiv:2508.18847](https://arxiv.org/abs/2508.18847) find
  37. Zhuohang Li, Xiaowei Li, Chengyu Huang, Guowang Li, Katayoon Goshvadi, Bo Dai, Dale Schuurmans, Paul Zhou, Hamid Palangi, Yiwen Song, Palash Goyal, Murat Kantarcioglu, Bradley A. Malin, and Yuan Xue. 2025b. Judging with confidence: Calibrating autoraters to preference distributions. arXiv preprint a… — [arXiv:2510.00263](https://arxiv.org/abs/2510.00263) find
  38. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv: 2205.14334. find
  39. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988.
  40. Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. 2020. Energy-based out-of-distribution detection. arXiv preprint arXiv: 2010.03759. — [arXiv:2010.03759](https://arxiv.org/abs/2010.03759) find
  41. Qing Lyu, Kumar Shridhar, Chaitanya Malaviya, Li Zhang, Yanai Elazar, Niket Tandon, Marianna Apidianaki, Mrinmaya Sachan, and Chris Callison-Burch. 2025. Calibrating large language models with sample consistency. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 1926… find
  42. Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896. find
  43. Mohammad Pezeshk Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence (AAAI), volume 29, pages 2901–2907. find
  44. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143, Online. Association for Computational Linguistics.
  45. Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. 2019. Measuring calibration in deep learning. arXiv preprint arXiv: 1904.01685. find
  46. OpenAI. 2024. Learning to reason with llms. OpenAI Blog. find
  47. OpenAI. 2025. gpt-oss-120b gpt-oss-20b model card. Preprint, arXiv:2508.10925.
  48. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  49. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694.
  50. Benjamin Plaut, Nguyen X. Khanh, and Tu Trinh. 2024. Probabilities of chat llms are miscalibrated but still predict correctness on multiple-choice qa. arXiv preprint arXiv: 2402.13213. — [arXiv:2402.13213](https://arxiv.org/abs/2402.13213) find
  51. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  52. Bhaktipriya Radharapu, Manon Revel, Megan Ung, Sebastian Ruder, and Adina Williams. 2025. Arbiters of ambivalence: Challenges of using llms in no-consensus tasks. arXiv preprint arXiv: 2505.23820. find
  53. Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. 2020. Mitigating bias in calibration error estimation. arXiv preprint arXiv: 2012.08668. — [arXiv:2012.08668](https://arxiv.org/abs/2012.08668) find
  54. Candace Ross, Melissa Hall, Adriana Romero-Soriano, and Adina Williams. 2024. What makes a good metric? evaluating automatic metrics for text-to-image consistency. In First Conference on Language Modeling. find
  55. Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. 2024. LLM-check: Investigating detection of hallucinations in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  56. Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. 2022. On the machine learning of ethical judgments from natural language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find
  57. Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, R. Popa, and Ion Stoica. 2024. Judgebench: A benchmark for evaluating llm-based judges. International Conference on Learning Representations. find
  58. Linwei Tao, Yi-Fan Yeh, Minjing Dong, Tao Huang, Philip Torr, and Chang Xu. 2025. Revisiting uncertainty estimation and calibration of large language models. arXiv preprint arXiv: 2505.23854. — [arXiv:2505.23854](https://arxiv.org/abs/2505.23854) find
  59. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conferen… find
  60. Zailong Tian, Zhuoheng Han, Yanzhe Chen, Haozhe Xu, Xi Yang, Richeng Xuan, Houfeng Wang, and Lizi Liao. 2025. Overconfidence in llm-as-a-judge: Diagnosis and confidence-driven solution. arXiv preprint arXiv: 2508.06225. — [arXiv:2508.06225](https://arxiv.org/abs/2508.06225) find
  61. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  62. Mark Tygert. 2025. Calibration and bias in algorithms, data, and models: a tutorial on metrics and plots for measuring calibration, bias, fairness, reliability, and robustness. find
  63. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. find
  64. Lily Hong Zhang, Smitha Milli, Karen Jusko, Jonathan Smith, Brandon Amos, Wassim Bouaziz, Manon Revel, Jack Kussman, Yasha Sheynin, Lisa Titus, and 1 others. 2025b. Cultivating pluralism in algorithmic monoculture: The community alignment dataset. arXiv preprint arXiv:2507.09650. — [arXiv:2507.09650](https://arxiv.org/abs/2507.09650) find
  65. Michael JQ Zhang, Zhilin Wang, Jena D. Hwang, Yi Dong, Olivier Delalleau, Yejin Choi, Eunsol Choi, Xiang Ren, and Valentina Pyatkin. 2024. Diverging preferences: When do annotators disagree and do models know? arXiv preprint arXiv: 2410.14632. find
  66. Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. International Conference on Machine Learning, pages 12697–12706.
  67. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.
  68. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv: 2203.11171.
  69. Yiming Wang, Pei Zhang, Baosong Yang, Derek F Wong, and Rui Wang. 2024. Latent space chain-of-embedding enables output-free llm self-evaluation. arXiv preprint arXiv:2410.13640. — [arXiv:2410.13640](https://arxiv.org/abs/2410.13640) find
  70. Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swarnadeep Saha. 2025. J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning. arXiv preprint arXiv: 2505.10320. — [arXiv:2505.10320](https://arxiv.org/abs/2505.10320) find
  71. Johnathan Xie, Annie S. Chen, Yoonho Lee, Eric Mitchell, and Chelsea Finn. 2024a. Calibrating language models with adaptive temperature scaling. arXiv preprint arXiv: 2409.19817. find
  72. Zhihui Xie, Jizhou Guo, Tong Yu, and Shuai Li. 2024b. Calibrating reasoning in language models with internal consistency. In Advances in Neural Information Processing Systems, volume 37, pages 114872– 114901. Curran Associates, Inc. find
  73. Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations. find
  74. Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine A Heller, and Subhrajit Roy. 2024a. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. In The Twelfth International Conference on Learning Representations. find
  75. Hongli Zhou, Hui Huang, Yunfei Long, Bing Xu, Conghui Zhu, Hailong Cao, Muyun Yang, and Tiejun Zhao. 2024b. Mitigating the bias of large language model evaluation. In Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference), pages 1310–1319, Taiyua… find
  76. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2025. Judgelm: Fine-tuned large language models are scalable judges. Preprint, arXiv:2310.17631.
  77. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, and 1 others. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.
  78. Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, and 1 others. 2024. Shieldgemma: Generative ai content moderation based on gemma. arXiv preprint arXiv:2407.21772. — [arXiv:2407.21772](https://arxiv.org/abs/2407.21772) find
  79. Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, and He He. 2025a. Reasoning models know when they’re right: Probing hidden states for self-verification. arXiv preprint arXiv: 2504.05419. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/radharapu-2026-calibrating-llm-judges-linear-probes/