Elamur Research⌕
☰
home / papers / 2026 / Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration

Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration

Paper · 2026 · Hang Lv, Hongchao Gu, Ruiqing Yang et al.

Summary

Generative listwise rerankers built on LLMs suffer from position bias: their output depends on where a document sits in the prompt, not only on its relevance. Current fixes either need many inference passes over shuffled lists, which is slow, or retraining, which is costly and may leave the bias in place for small models. The authors propose CapCal, a training-free calibration method. It runs the frozen model once more on the same prompt, keeping the query and document identifiers but leaving every passage empty. The probabilities the model assigns to identifiers in this content-free setting serve as an estimate of its positional prior. CapCal then scores each identifier at the level of its full token sequence and subtracts this prior from the normal ranking probabilities. The subtraction uses a contrastive scheme whose strength adapts to entropy. Across 10 benchmarks from MS MARCO and BEIR, and on Qwen models from 0.6B to 8B, the method improves NDCG@10 while adding only one extra forward pass. The gains are largest for the 0.6B model. This makes compact rerankers more practical when latency matters.

Key points

  • Benchmarks. TREC DL19–DL23 cover the MS MARCO v1 and v2 corpora. Five BEIR datasets cover specialized domains: TREC-COVID, NFCorpus, Climate-FEVER, FiQA and Arguana. The authors treat DL21 results as reference only.

  • Models and average NDCG@10 (in-domain / out-of-domain):

    • Qwen3-0.6B: 0.4985 → 0.5493 and 0.2969 → 0.3560.
    • Qwen3-8B: 0.6675 → 0.6870 and 0.4061 → 0.4177.
    • Qwen2.5-7B-Instruct: 0.6603 → 0.6623 and 0.3950 → 0.4157. This model drops slightly on a few individual sets.

  • Largest gains. For Qwen3-0.6B, single-dataset improvements exceed 10 NDCG@10 points. The paper reports this for MS MARCO V2 and Climate-FEVER.

  • Versus Permutation Self-Consistency (PSC). PSC uses 10 shuffled passes per query. On Qwen3-0.6B, CapCal matches or beats it with just one additional pass.

  • Robustness and augmentation. Gains persist across several changes:

    • a stronger first-stage retriever (bge-reranker-v2-m3);
    • randomly shuffled candidate lists;
    • different placeholder content and length;
    • letter identifiers instead of numerals.

    Fine-tuning Qwen3-0.6B with RankZephyr-style permutation augmentation still leaves substantial positional bias. Code is available at USTC-StarTeam/CapCal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China201 works · 17,172 citations🎓University of Science and Technology of ChinaHefei · China145 works · 114,409 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

Cite

Cite as Hang Lv, Hongchao Gu, Ruiqing Yang, Liangyue Li, Zulong Chen, Defu Lian, Hao Wang and Enhong Chen. 2026. Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.68

@inproceedings{lv2026learning,
  title = {{Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration}},
  author = {Hang Lv, Hongchao Gu, Ruiqing Yang, Liangyue Li, Zulong Chen, Defu Lian, Hao Wang and Enhong Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.68},
  url = {https://doi.org/10.18653/v1/2026.acl-short.68}
}

References

38 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. Preprint, arXiv:2402.03216.
  2. Shijie Chen, Bernal Jiménez Gutiérrez, and Yu Su. 2025. Attention in large language models yields efficient zero-shot re-rankers. Preprint, arXiv:2410.02642. — [arXiv:2410.02642](https://arxiv.org/abs/2410.02642) find
  3. Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2024. Dola: Decoding by contrasting layers improves factuality in large language models. Preprint, arXiv:2309.03883.
  4. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the trec 2020 deep learning track. Preprint, arXiv:2102.07662. — [arXiv:2102.07662](https://arxiv.org/abs/2102.07662) find
  5. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2025a. Overview of the trec 2021 deep learning track. Preprint, arXiv:2507.08191. — [arXiv:2507.08191](https://arxiv.org/abs/2507.08191) find
  6. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2025b. Overview of the trec 2022 deep learning track. Preprint, arXiv:2507.10865. — [arXiv:2507.10865](https://arxiv.org/abs/2507.10865) find
  7. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. Overview of the trec 2019 deep learning track. Preprint, arXiv:2003.07820.
  8. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Hossein A. Rahmani, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2025c. Overview of the trec 2023 deep learning track. Preprint, arXiv:2507.08890. — [arXiv:2507.08890](https://arxiv.org/abs/2507.08890) find
  9. Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. 2024. FIRST: Faster improved listwise reranking with single token decoding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8642–8652, Miami, Florida… find
  10. Hongchao Gu, Dexun Li, Kuicai Dong, Hao Zhang, Hang Lv, Hao Wang, Defu Lian, Yong Liu, and Enhong Chen. 2025. Rapid: Efficient retrieval-augmented long text generation with writing planning and information discovery. Preprint, arXiv:2503.00751. 822 6 find
  11. Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T. Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. 2024. Found in the middle: Calibrating positional attention bias improves long context utilization. Preprint, arXiv:2406.16008. find
  12. Youngwon Lee, Seung-won Hwang, Daniel F Campos, Filip Graliński, Zhewei Yao, and Yuxiong He. 2025. Inference scaling for bridging retrieval and augmented generation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7324–7339, Albuquerque, New Mexico. Association for C… find
  13. Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2023. Contrastive decoding: Open-ended text generation as optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: L…
  14. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  15. Wenhan Liu, Xinyu Ma, Weiwei Sun, Yutao Zhu, Yuchen Li, Dawei Yin, and Zhicheng Dou. 2025. Reasonrank: Empowering passage ranking with strong reasoning ability. Preprint, arXiv:2508.07050.
  16. Hang Lv, Sheng Liang, Hao Wang, Hongchao Gu, Yaxiong Wu, Wei Guo, Defu Lian, Yong Liu, and Enhong Chen. 2026a. Costeer: Collaborative decoding-time personalization via local delta steering. Preprint, arXiv:2507.04756. — [arXiv:2507.04756](https://arxiv.org/abs/2507.04756) find
  17. Hang Lv, Sheng Liang, Hao Wang, Yongyue Zhang, Hongchao Gu, Wei Guo, Defu Lian, Yong Liu, and Enhong Chen. 2026b. Specsteer: Synergizing local context and global reasoning for efficient personalized generation. Preprint, arXiv:2603.16219. — [arXiv:2603.16219](https://arxiv.org/abs/2603.16219) find
  18. Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero-shot listwise document reranking with a large language model. Preprint, arXiv:2305.02156.
  19. Sean O’Brien and Mike Lewis. 2023. Contrastive decoding improves reasoning in large language models. Preprint, arXiv:2309.09117.
  20. Keqin Peng, Liang Ding, Yuanxin Ouyang, Meng Fang, Yancheng Yuan, and Dacheng Tao. 2025. Enhancing input-label mapping in in-context learning with contrastive decoding. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 997–1004… find
  21. Phuc Phan, Hieu Tran, and Long Phan. 2024. Distillation contrastive decoding: Improving llms reasoning with contrastive decoding and distillation. Preprint, arXiv:2402.14874.
  22. Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023a. RankVicuna: Zero-shot listwise document reranking with open-source large language models. arXiv:2309.15088.
  23. Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023b. RankZephyr: Effective and robust zero-shot listwise reranking is a breeze! arXiv:2312.02724. — [arXiv:2312.02724](https://arxiv.org/abs/2312.02724) find
  24. Zexuan Qiu, Zijing Ou, Bin Wu, Jingjing Li, Aiwei Liu, and Irwin King. 2025. Entropy-based decoding for retrieval-augmented large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologie… find
  25. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  26. Ruiyang Ren, Yuhao Wang, Kun Zhou, Wayne Xin Zhao, Wenjie Wang, Jing Liu, Ji-Rong Wen, and Tat-Seng Chua. 2024. Self-calibrated listwise reranking with large language models. Preprint, arXiv:2411.04602. — [arXiv:2411.04602](https://arxiv.org/abs/2411.04602) find
  27. Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. 2024. Trusting your evidence: Hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language…
  28. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1491…
  29. Raphael Tang, Crystina Zhang, Xueguang Ma, Jimmy Lin, and Ferhan Ture. 2024. Found in the middle: Permutation self-consistency improves listwise ranking in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hum… find
  30. Papers), pages 2327–2340, Mexico City, Mexico. Association for Computational Linguistics. find
  31. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023. Zephyr: Direct distillation of lm alignment. Preprint, arXiv… find
  32. Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, and Benjamin Van Durme. 2025. Rank1: Test-time compute for reranking in information retrieval. Preprint, arXiv:2502.18418. — [arXiv:2502.18418](https://arxiv.org/abs/2502.18418) find
  33. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388.
  34. Eugene Yang, Andrew Yates, Kathryn Ricci, Orion Weller, Vivek Chari, Benjamin Van Durme, and Dawn Lawrie. 2025b. Rank-k: Test-time reasoning for listwise reranking. Preprint, arXiv:2505.14432. — [arXiv:2505.14432](https://arxiv.org/abs/2505.14432) find
  35. Soyoung Yoon, Eunbi Choi, Jiyeon Kim, Hyeongu Yun, Yireun Kim, and Seung-won Hwang. 2024. ListT5: Listwise reranking with fusion-in-decoder improves zero-shot retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2287– 23… find
  36. Yifan Zeng, Ojas Tendolkar, Raymond Baartmans, Qingyun Wu, Lizhong Chen, and Huazheng Wang. 2024. Llm-rankfusion: Mitigating intrinsic inconsistency in llm-based ranking. Preprint, arXiv:2406.00231. — [arXiv:2406.00231](https://arxiv.org/abs/2406.00231) find
  37. Zheng Zhao, Emilio Monti, Jens Lehmann, and Haytham Assem. 2024. Enhancing contextual understanding in large language models through contrastive decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… find
  38. Xuyang Zhi, Peilun zhou, Chengqiang Lu, Hang Lv, Yiwei Liang, Rongyang Zhang, Yan Gao, YI WU, Yao Hu, Hongchao Gu, Defu Lian, Hao Wang, and Enhong Chen. 2026. Spard: Self-paced curriculum for rl alignment via integrating reward dynamics and data utility. Preprint, arXiv:2604.07837.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lv-2026-learning-emptiness-de-biasing-listwise/