Elamur Research⌕
☰
home / papers / 2026 / Index-Time Prefix Injection for Multi-Tenant Retrieval: Improving Search Relevance Without Model Fine-Tuning

Index-Time Prefix Injection for Multi-Tenant Retrieval: Improving Search Relevance Without Model Fine-Tuning

Paper · 2026 · Vaibhav Varshney and Manjunatha Naik Mc

Summary

Enterprise search platforms that serve many customers usually rely on one shared retrieval encoder. Contractual data isolation prevents fine-tuning it for each customer, and separate per-customer models do not scale. The authors (ServiceNow) propose index-time prefix injection, which requires no training. Before encoding, each document is prepended with a short natural-language description of its datasource. A frozen instruction-tuned bi-encoder (E5-style) then places that datasource's embeddings in a more domain-coherent region. Queries keep only the model's standard prefix, so a single query encoding can search several datasources at once.

Prefixes come from a three-tier pipeline:

  • generation by an LLM from sampled documents, where data policy permits;
  • descriptions written by customer experts;
  • a standardized multilingual library as the fallback.

The system was evaluated offline on 10K annotated production queries and online in a 4-week A/B test across 400+ instances in 18 languages. It improved Hit@5 by 3–8% and CTR by 4.2%. Query latency was unchanged. The result shows a cheap, encoder-agnostic way to adapt retrieval when fine-tuning is not allowed.

Key points

  • Tier 1 (LLM-generated prefixes): samples 5 documents and produces 5 candidate prefixes of 8–15 words, then picks one at random. Random choice matched click-based and synthetic-query selection. GPT-4o, GPT-4o-mini, Claude Sonnet and Claude Haiku performed within 0.3% Hit@5 of each other.
  • Tier 3 (standardized library): 30 prefixes, each available in 18 languages. A fastText classifier assigns each datasource to a prefix with 87% top-1 accuracy.
  • Prefix design: colon-terminated prefixes in the document's own language worked best. Datasources with fewer than 50 documents gained almost nothing.
  • A/B test results: CTR rose from 62.3% to 64.9%, the null-result rate fell by 5.7%, and clicks per session rose by 7.8% (p < 0.01). Indexing became 7% slower, while query latency stayed unchanged.
  • Downstream RAG QA: human-judged answer correctness improved by 4.7 points on IT KB articles and 5.6 points on product catalog items. Answer attribution also rose.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Vaibhav Varshney and Manjunatha Naik Mc. 2026. Index-Time Prefix Injection for Multi-Tenant Retrieval: Improving Search Relevance Without Model Fine-Tuning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.149

@inproceedings{varshney2026index,
  title = {{Index-Time Prefix Injection for Multi-Tenant Retrieval: Improving Search Relevance Without Model Fine-Tuning}},
  author = {Vaibhav Varshney and Manjunatha Naik Mc},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.149},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.149}
}

References

26 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. The Claude 3 model family: A new standard for intelligence. find
  2. Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2023. Task-aware retrieval with instructions. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3650– 3675, Toronto, Canada.
  3. Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 39–48.
  4. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. NV-Embed: Improved techniques for training LLMs as generalist embedding models. arXiv preprint arXiv:2405.17428.
  5. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898– 2904.
  6. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059.
  7. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. BGE M3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216.
  8. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… find
  9. Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise zero-shot dense retrieval without relevance labels. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1762–1777.
  10. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare, 3(1):1–23.
  11. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360. find
  12. Helia Hashemi, Hamed Zamani, and W. Bruce Croft. 2023. Dense retrieval adaptation using target domain description. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR), pages 95–104, Taipei, Taiwan. find
  13. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research (TMLR).
  14. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781.
  15. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.
  16. Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2024. Generative representational instruction tuning. arXiv preprint arXiv:2402.09906.
  17. Hanseok Oh, Hyunji Seo, Minjin Zhong, and 1 others. 2024. InstructIR: A benchmark for instruction following of information retrieval models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. find
  18. OpenAI. 2024. GPT-4o system card. arXiv preprint arXiv:2410.21276.
  19. Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023. One embedder, any task: Instruction-finetuned text embeddings. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1102–1121, Toronto…
  20. Zhengyang Tang, Benyou Wang, and Ting Yao. 2022. DPTDR: Deep prompt tuning for dense passage retrieval. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1193– 1202. find
  21. Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022a. GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
  22. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022b. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.
  23. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024a. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916, Bangkok, Thailand.
  24. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024b. Multilingual E5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672.
  25. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2024. C-Pack: Packaged resources to advance general Chinese embedding. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Washington, DC, USA. find
  26. Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. mGTE: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/varshney-2026-index-time-prefix-injection-multi/