Elamur Research⌕
☰
home / papers / 2026 / Confidence-Aware Ranker Ensembles for Robust In-Context Knowledge Editing

Confidence-Aware Ranker Ensembles for Robust In-Context Knowledge Editing

Paper · 2026 · Tejal Nair, Mahmud Wasif Nafee, Maiqi Jiang et al.

Summary

Large language models can keep producing outdated or wrong facts. In-context knowledge editing (IKE) addresses this at inference time by putting corrected facts and demonstrations into the prompt, without changing weights, which makes it suitable for black-box APIs. However, current IKE pipelines pick demonstrations with a single retriever whose scores are not calibrated. The authors find that different rankers surface different useful evidence, and that the most reliable one varies from query to query. They propose FWE-IKE, which treats LLM-, BERT- and MLP-based rankers as experts, in the spirit of mixture-of-experts. Each ranker's score distribution is first made comparable through top-K truncation and a temperature tuned to a target entropy. Cheap confidence features then feed a small meta-model that predicts per-query weights for fusion in logit space. When one expert clearly dominates, a cautious margin gate routes the query to it alone. On CounterFact, the ensemble outperforms every individual retriever and comes close to the oracle ceiling. This suggests adaptive ranker combination makes gradient-free editing more robust.

Key points

  • On a 300-query CounterFact evaluation, FWE-IKE reaches 88.33% Edit-Success Rate. This beats BERT (85.33%), LLM (84.00%) and MLP (82.67%) rankers, a +3.0 point gain over the best one, against a 91.00% oracle bound.
  • The confidence features are normalized entropy, top-1 margin, top-1 mass and cross-retriever agreement. They are computed from each ranker's raw scores over a shared COPY/UPDATE/RETAIN candidate pool.
  • In ablations, both score calibration and logit-space fusion help, and the small gate adds robustness at little cost.
  • Experiments use Llama-3.2-3B because of GPU limits. Some baseline comparisons were run on reduced samples.
  • Stated limitations:
    • gains depend on the experts actually disagreeing;
    • only a single benchmark is used;
    • the gate relies on shallow signals that may fail out of distribution.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Rensselaer Polytechnic InstituteTroy · United States21 works · 4,933 citations🎓Bangladesh University of Engineering and TechnologyDhaka · Bangladesh7 works · 308 citations🎓William & MaryWilliamsburg · United States5 works · 357 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Tejal Nair, Mahmud Wasif Nafee, Maiqi Jiang, Ashley Gao, Haipeng Chen and Yanfu Zhang. 2026. Confidence-Aware Ranker Ensembles for Robust In-Context Knowledge Editing. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1750

@inproceedings{nair2026confidence,
  title = {{Confidence-Aware Ranker Ensembles for Robust In-Context Knowledge Editing}},
  author = {Tejal Nair, Mahmud Wasif Nafee, Maiqi Jiang, Ashley Gao, Haipeng Chen and Yanfu Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1750},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1750}
}

References

30 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Keqin Bao, Ming Yan, Yang Zhang, Jizhi Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2024. Real-time personalization for llm-based recommendation with customized in-context learning. find
  2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  3. Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2023. Evaluating the ripple effects of knowledge editing in language models. find
  4. Gordon Cormack, Charles Clarke, and Stefan Büttcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. pages 758–759.
  5. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. find
  6. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Preprint, arXiv:2101.03961. find
  7. Yujie Feng, Zexin Lu, Bo Liu, Liming Zhan, and Xiao-Ming Wu. 2023. Towards llm-driven dialogue state tracking. find
  8. Runyu He, Anyu Ying, and Xiaoyu Hu. 2024. Improving opendevin: Boosting code generation llm through advanced memory management. Applied and Computational Engineering, 68:313–320. find
  9. Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. 1991. Adaptive mixtures of local experts. Neural Computation, 3:79–87. find
  10. Michael Jordan and Robert Jacobs. 1994. Hierarchical mixtures of experts and the. Neural computation, 6:181–. find
  11. To Eun Kim and Fernando Diaz. 2025. Ltrr: Learning to rank retrievers for llms. Preprint, arXiv:2506.13743. — [arXiv:2506.13743](https://arxiv.org/abs/2506.13743) find
  12. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. Preprint, arXiv:2006.16668. find
  13. Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023. Unified demonstration retriever for in-context learning.
  14. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3?
  15. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. Preprint, arXiv:2104.08786.
  16. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2023. Locating and editing factual associations in gpt. find
  17. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023a. Mass-editing memory in a transformer. find
  18. Dehai Min, Nan Hu, Rihui Jin, Nuo Lin, Jiaoyan Chen, Yongrui Chen, Yu Li, Guilin Qi, Yun Li, Nijun Li, and Qianren Wang. 2024. Exploring the impact of table-to-text methods on augmenting llm-based question answering with domain hybrid data. find
  19. Shuyang Yu, Runxue Bao, Parminder Bhatia, Taha Kass-Hout, Jiayu Zhou, and Cao Xiao. 2025. Dynamic uncertainty ranking: Enhancing retrieval-augmented in-context learning for long-tail knowledge in llms. find
  20. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. Preprint, arXiv:2102.09690.
  21. Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. 2023a. Can we edit factual knowledge by in-context learning? Preprint, arXiv:2305.12740. find
  22. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023b. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.
  23. Chenghao Zhu, Nuo Chen, Yufei Gao, Yunyi Zhang, Prayag Tiwari, and Benyou Wang. 2025. Is your llm outdated? a deep look at temporal generalization. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologi… find
  24. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022. Fast model editing at scale. find
  25. Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. 2024. Dyknow: Dynamically verifying time-sensitive factual knowledge in llms. find
  26. Mahmud Wasif Nafee, Maiqi Jiang, Haipeng Chen, and Yanfu Zhang. 2025. Dynamic retriever for in-context knowledge editing via policy optimization. Preprint, arXiv:2510.21059. find
  27. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. volume 3, pages 333–389. find
  28. Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. 2023. Prompting GPT-3 to be reliable. In The Eleventh International Conference on Learning Representations.
  29. Changyue Wang, Weihang Su, Qingyao Ai, Yichen Tang, and Yiqun Liu. 2025. Knowledge editing through chain-of-thought. find
  30. Liang Wang, Nan Yang, and Furu Wei. 2024. Learning to retrieve in-context examples for large language models.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nair-2026-confidence-aware-ranker-ensembles-robust/