Elamur Research⌕
☰
home / papers / 2026 / Situated Embedding Models for Context-Aware Dense Retrieval

Situated Embedding Models for Context-Aware Dense Retrieval

Paper · 2026 · Junjie Wu, Jiangnan Li, Yuqing Li et al.

Summary

When RAG systems split long documents into chunks, a chunk often cannot be interpreted correctly without the surrounding text. Feeding longer chunks to long-context embedders does not solve this. On a book-plot retrieval task, recall falls as chunk size grows, because one vector must compress more information. The authors instead keep retrieval units short and condition each chunk's embedding on its neighbouring text, which they call situated embedding. They find that current models lose accuracy when given such context. To train the first dedicated situated embedding model, they use user notes from Douban anchored to book passages, with the user-underlined span serving as context. A residual design pairs a chunk-only encoder with a context-aware encoder whose outputs are summed, which pushes the second model to use the context rather than surface clues in the chunk. On a new Book Plot Retrieval benchmark built from Chinese book translations, the 1B Sit-M3 beats 7B embedders such as NV-Embed-v2. This shows that targeted training matters more than model scale for context-dependent retrieval.

Key points

  • Training data: 1,614,007 note–anchor pairs from about 100 popular Douban books. Context lengths range from 37 tokens to several thousand.
  • Model and training: Sit-M3 is built from two BGE-M3 encoders (0.5B each, about 1B total). It uses a margin-based loss with 10 negatives drawn from other chapters of the same book.
  • Benchmark: Book Plot Retrieval is derived from PlotRetrieval and covers 7 books with 1,394 queries. Situated contexts span 2,048–6,144 tokens.
  • Results on NDP-v1: Adding context makes baselines collapse; for example, M3 drops from 42.55 to 9.48 R@10. Sit-M3 instead reaches 51.73 R@10 and 75.00 R@50, compared with 71.16 R@50 for NV-Embed-v2.
  • 7-book average and transfer: Sit-M3 scores 44.71/56.16/69.53 (R@10/20/50), versus 42.87/52.91/66.01 for trained M3. Removing the residual design lowers scores, and a chunk-only residual model (Res-M3) does not help. Sit-M3 also yields small gains on Recap Snippet Identification, LoCoV1 and LongStoryQA, raising Ans-F1 from 52.24 to 53.91. The models are released at huggingface.co/SituatedEmbedding.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 552 works

Cite

Cite as Junjie Wu, Jiangnan Li, Yuqing Li, Lemao Liu, Liyan Xu, Jiwei Li, Dit-Yan Yeung, Jie Zhou and Mo Yu. 2026. Situated Embedding Models for Context-Aware Dense Retrieval. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.5

@inproceedings{wu2026situated,
  title = {{Situated Embedding Models for Context-Aware Dense Retrieval}},
  author = {Junjie Wu, Jiangnan Li, Yuqing Li, Lemao Liu, Liyan Xu, Jiwei Li, Dit-Yan Yeung, Jie Zhou and Mo Yu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.5},
  url = {https://doi.org/10.18653/v1/2026.acl-short.5}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. Preprint, arXiv:2402.03216.
  2. Michael Günther, Isabelle Mohr, Daniel James Williams, Bo Wang, and Han Xiao. 2024. Late chunking: contextual chunk embeddings using long-context embedding models. arXiv preprint arXiv:2409.04701. — [arXiv:2409.04701](https://arxiv.org/abs/2409.04701) find
  3. Anthropic. 2024. Enhancing rag with contextual retrieval. find
  4. Allyson Ettinger. 2020. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. \#\# Conclusion This paper introduces the first situated embedding model designed to incorporate a chunk’s con…
  5. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778.
  6. Xinshuo Hu, Zifei Shan, Xinping Zhao, Zetian Sun, Zhenyu Liu, Dongfang Li, Shaolin Ye, Xinyuan Wei, Qian Chen, Baotian Hu, and 1 others. 2025. Kalm-embedding: Superior training data brings a stronger embedding model. arXiv preprint arXiv:2501.01028. \#\# Limitations This work is the first to train a s… — [arXiv:2501.01028](https://arxiv.org/abs/2501.01028) find
  7. Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. One thousand and one pairs: A "novel" challenge for long-context language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 202… find
  8. Jihoon Kwon Sangmo Gu Yejin Kim, Minkyung Cho Jy-yong Sohn Chanyeol, Choi Junseong Kim, and Seolhwa Lee. 2024. Linq-embed-mistral: Elevating text retrieval with improved gpt data through task-specific control and quality refinement. linq ai research blog. find
  9. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328.
  10. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428. \#\# Acknowledgment This work has been made possible by a Research Impact Fund…
  11. Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Andreas Koukounas, Nan Wang, and Han Xiao. 2024. jina-embeddings-v3: Multilingual embeddings with task lora. Preprint, arXiv:2409.10173.
  12. Jiangnan Li, Qiujing Wang, Liyan Xu, Wenjie Pang, Mo Yu, Zheng Lin, Weiping Wang, and Jie Zhou. 2024. Previously on the stories: Recap snippet identification for story reading. arXiv preprint arXiv:2402.07271. — [arXiv:2402.07271](https://arxiv.org/abs/2402.07271) find
  13. Voyage-AI. 2025. Introducing voyage-context-3: focused chunk-level details with global document context. Blog post. find
  14. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.
  15. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024a. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916.
  16. Raiza Martin and Steven Johnson. 2023. Introducing notebooklm. find
  17. Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. 2024b. Coderag-bench: Can retrieval augment code generation? arXiv preprint arXiv:2406.14497. find
  18. Jing Miao, Charat Thongprayoon, Supawadee Sup-padungsuk, Oscar A Garcia Valencia, and Wisit Che-ungpasitporn. 2024. Integrating retrieval-augmented generation with large language models in nephrology: advancing practical applications. Medicina, 60(3):445. find
  19. Liyan Xu, Jiangnan Li, Mo Yu, and Jie Zhou. 2024a. Fine-grained modeling of narrative context: A coherence perspective via retrospective questions. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5822–5838. find
  20. Gabriel de Souza P Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. 2024. Nv-retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint arXiv:2407.15831. — [arXiv:2407.15831](https://arxiv.org/abs/2407.15831) find
  21. Shicheng Xu, Liang Pang, Jiangnan Li, Mo Yu, Fandong Meng, Huawei Shen, Xueqi Cheng, and Jie Zhou. 2024b. Plot retrieval as an assessment of abstract semantic association. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)… find
  22. Xiangyang Mou, Chenghao Yang, Mo Yu, Bingsheng Yao, Xiaoxiao Guo, Saloni Potdar, and Hui Su. 2021. Narrative question answering with cutting-edge open-domain qa techniques: A comprehensive study. Transactions of the Association for Computational Linguistics, 9:1032–1046. find
  23. Zhe Xu, Jiasheng Ye, Xiaoran Liu, Xiangyang Liu, Tianxiang Sun, Zhigeng Liu, Qipeng Guo, Linlin Li, Qun Liu, Xuanjing Huang, and Xipeng Qiu. 2025. DetectiveQA: Evaluating long-context reasoning on detective novels. In Workshop on Reasoning and Planning for Large Language Models. find
  24. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037.
  25. Zach Nussbaum, John X Morris, Brandon Duderstadt, and Andriy Mulyar. 2024. Nomic embed: Training a reproducible long context text embedder. arXiv preprint arXiv:2402.01613.
  26. Mo Yu, Jiangnan Li, Shunyu Yao, Wenjie Pang, Xiaochen Zhou, Zhou Xiao, Fandong Meng, and Jie Zhou. 2023. Personality understanding of fictional characters during book reading. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14… find
  27. OpenAI. 2024. Hello gpt-4o. find
  28. Zexuan Qiu, Jingjing Li, Shijue Huang, Xiaoqi Jiao, Wanjun Zhong, and Irwin King. 2024. Clongeval: A chinese benchmark for evaluating long-context large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3985–4004. find
  29. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024. ∞bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguist… find
  30. Jon Saad-Falcon, Daniel Y Fu, Simran Arora, Neel Guha, and Christopher Ré. 2024. Benchmarking and building long-context retrieval models with loco Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou.…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-situated-embedding-models-context-aware/