Elamur Research⌕
☰
home / papers / 2026 / Toward Robust In-Context Learning: Leveraging Out-of-distribution Proxies for Target Inaccessible Demonstration Retrieval

Toward Robust In-Context Learning: Leveraging Out-of-distribution Proxies for Target Inaccessible Demonstration Retrieval

Paper · 2026 · Hao Xu, Rite Bo, Fausto Giunchiglia et al.

Summary

In-context learning with LLMs loses accuracy when test inputs come from a distribution different from the pool of labelled examples. Picking good demonstrations is especially hard when no target-domain samples are available. The authors present DOPA, a retrieval framework that relies only on source-domain data.

DOPA builds an OOD proxy from two copies of one LLM. The "source proxy" is instruction-tuned on the source domain. The untouched base model acts as a stand-in for the unknown target distribution, since its broad pretraining is assumed to carry relevant prior knowledge. The ratio of perplexities the two models assign to a source sample gives an OOD score, which estimates how target-like that sample is. This score is combined with embedding similarity to the test input to shortlist candidates. A Mahalanobis-distance constraint then keeps the final demonstration set varied.

A bounded proxy-error analysis provides theoretical support. Across several LLMs and NLU tasks, DOPA outperforms retrieval baselines such as KNN. This makes ICL more dependable in deployments where target data cannot be inspected beforehand.

Key points

  • Proxy design: the OOD score is the perplexity ratio between a source-fine-tuned LLM and its base version. This replaces the uniform-distribution target assumption used in earlier OOD detection work, which the authors argue is suboptimal.
  • Evaluation setup: tasks are labelled SA, NLI, NER and TD. Models analysed include LLaMA3.2-3B and Qwen3-1.7B.
  • Hyperparameters:
    • The candidate pool size k was tested at 300, 500, 800 and 1000. The authors fixed k = 800 as a balance between diversity and noise.
    • The per-label demonstration count N was varied from 1 to 5 (N × |Y| examples in total) and fixed at 3 for the main experiments.
    • DOPA beat KNN at every N. In SA, performance peaked at N = 4 for Qwen3-1.7B and N = 3 for LLaMA3.2-3B.
  • Visualization on sst:
    • KDE plots of BERT-based energy scores show that proxy-selected samples overlap the target distribution.
    • t-SNE plots place the same samples near the source domain, suggesting that semantic similarity alone is insufficient for OOD retrieval.
    • The Mahalanobis constraint increases diversity only moderately, without large semantic drift.
  • Code: available at github.com/bort64/ood_code (ACL 2026 long paper, Jilin University and University of Trento).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksEngineering · 188 worksControl and Systems Engineering · 62 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksInformation retrieval & search · 552 works

Cite

Cite as Hao Xu, Rite Bo, Fausto Giunchiglia, Yingji Li and Rui Song. 2026. Toward Robust In-Context Learning: Leveraging Out-of-distribution Proxies for Target Inaccessible Demonstration Retrieval. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.941

@inproceedings{xu2026toward,
  title = {{Toward Robust In-Context Learning: Leveraging Out-of-distribution Proxies for Target Inaccessible Demonstration Retrieval}},
  author = {Hao Xu, Rite Bo, Fausto Giunchiglia, Yingji Li and Rui Song},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.941},
  url = {https://doi.org/10.18653/v1/2026.acl-long.941}
}

References

38 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. In-context examples selection for machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 8857–8873.
  2. Haoyue Bai, Xuefeng Du, Katie Rainey, Shibin Parameswaran, and Yixuan Li. 2024. Out-of-distribution learning with human feedback. CoRR, abs/2408.07772. — [arXiv:2408.07772](https://arxiv.org/abs/2408.07772) find
  3. Christopher M Bishop. 1993. Novelty detection and neural network validation. In ICANN’93: Proceedings of the International Conference on Artificial Neural Networks Amsterdam, The Netherlands 13– 16 September 1993 3, pages 789–794. Springer. find
  4. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol., 15(3):39:1– 39:45.
  5. Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Weiwei Deng, and Qi Zhang. 2023. UPRISE: universal prompt retrieval for improving zero-shot evaluation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP… find
  6. Janez Demsar. 2006. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res., 7:1–30. find
  7. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 6894– 6910.
  8. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
  9. Shuoran Jiang, Qingcai Chen, Yang Xiang, Youcheng Pan, and Yukang Lin. 2024. Linguistic rule induction improves adversarial and OOD robustness in large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COL… find
  10. Janak Kapuriya, Manit Kaushik, Debasis Ganguly, and Sumit Bhatia. 2025. Exploring the role of diversity in example selection for in-context learning. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2025, Padua, Italy, July 13-… find
  11. Imbrasaite, and Vincent Y. Zhao. 2023. Dr.icl: Demonstration-retrieved in-context learning. CoRR, abs/2305.14128. — [arXiv:2305.14128](https://arxiv.org/abs/2305.14128) find
  12. Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi. 2024. In-context learning with retrieved demonstrations for language models: A survey. CoRR, abs/2401.11624.
  13. Manas Madine. 2024. Bridging distribution gap via semantic rewriting with llms to enhance OOD robustness. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 - Student Research Workshop, Bangkok, Thailand, August 11-16, 2024, pages 458–468. find
  14. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi…
  15. Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, and Balaji Lakshminarayanan. 2019. Do deep generative models know what they don’t know? In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. find
  16. Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023a. Unified demonstration retriever for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toro…
  17. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and 1 others. 2022. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844– 98…
  18. Yingji Li, Mengnan Du, Xin Wang, and Ying Wang. 2023b. Prompt tuning pushes farther, contrastive learning pulls closer: A two-stage approach to mitigate social biases. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toront… find
  19. Kyle O’Brien, Nathan Ng, Isha Puri, Jorge Mendez, Hamid Palangi, Yoon Kim, Marzyeh Ghassemi, and Thomas Hartvigsen. 2024. Improving black-box robustness with in-context rewriting. CoRR, abs/2402.08225. find
  20. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO@ACL 2022, Dublin, Ir…
  21. Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. 2020. Energy-based out-of-distribution detection. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. find
  22. Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Seyed Mehran Kazemi, Chitta Baral, Vaiva Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2024. Revisiting demonstration selection strategies in in-context learning. In Proceedings of the 62nd Annual Meeting… find
  23. Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, and Balaji Lakshminarayanan. 2019. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Syst… find
  24. Vinay M. S., Minh-Hao Van, and Xintao Wu. 2024. In-context learning demonstration selection via influence analysis. CoRR, abs/2402.11750. — [arXiv:2402.11750](https://arxiv.org/abs/2402.11750) find
  25. Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Yinxiao Liu, Simon Tong, Jindong Chen, and Lei Meng. 2024. Rewritelm: An instruction-tuned large language model for text rewriting. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applic… find
  26. Rui Song, Fausto Giunchiglia, Yingji Li, Mingjie Tian, and Hao Xu. 2024a. TACIT: A target-agnostic feature disentanglement framework for cross-domain text classification. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, pages 18999–19007. find
  27. Rui Song, Yingji Li, Lida Shi, Fausto Giunchiglia, and Hao Xu. 2024b. Shortcut learning in in-context learning: A survey. CoRR, abs/2411.02018. — [arXiv:2411.02018](https://arxiv.org/abs/2411.02018) find
  28. Rui Song, Yingji Li, Mingjie Tian, Hanwen Wang, Fausto Giunchiglia, and Hao Xu. 2025. Causal keyword driven reliable text classification with large language model feedback. Inf. Process. Manag., 62(3):103964. on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proce… find
  29. Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023. Revisiting out-of-distribution robustness in NLP: benchmarks, analysis, and llms evaluations. In Advances in Neural Information Processing Systems 36: Annual Conference on Neur… find
  30. Andi Zhang and Damon Wischik. 2022. Falsehoods that ML researchers believe about OOD detection. In NeurIPS ML Safety Workshop. find
  31. Andi Zhang, Tim Z. Xiao, Weiyang Liu, Robert Bamler, and Damon Wischik. 2024. Your finetuned large language model is already a powerful out-of-distribution detector. CoRR, abs/2404.08679. — [arXiv:2404.08679](https://arxiv.org/abs/2404.08679) find
  32. Rheeya Uppaal, Junjie Hu, and Yixuan Li. 2023. Is fine-tuning needed? pre-trained language models are near perfect for out-of-domain detection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 20… find
  33. Liang Wang, Nan Yang, and Furu Wei. 2024. Learning to retrieve in-context examples for large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 - Volume 1: Long Papers, St. Julian’s, Malta, March 17-22, 2024, pag…
  34. Qixun Wang, Yifei Wang, Xianghua Ying, and Yisen Wang. 2025. Can in-context learning really generalize to out-of-distribution tasks? In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. find
  35. Stanislaw Wkeglarczyk. 2018. Kernel density estimation and its application. In ITM web of conferences, volume 23, page 00037. EDP Sciences. find
  36. Arthur Wuhrmann, Andrei Kucharavy, and Anastasiia Kucherenko. 2025. Low-perplexity llm-generated sequences and where to find them. In ACL 2025 Student Research Workshop. find
  37. Shangqing Xu and Chao Zhang. 2024. Misconfidence-based demonstration selection for LLM in-context learning. CoRR, abs/2401.06301. — [arXiv:2401.06301](https://arxiv.org/abs/2401.06301) find
  38. Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In International Conference

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-robust-context-learning-leveraging-out/