Elamur Research⌕
☰
home / papers / 2026 / MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning

MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning

Paper · 2026 · Vera Pavlova and Mohammed Makhlouf

Summary

General-purpose text embedding models are pretrained on huge corpora that specialized fields cannot match, so they often lack domain knowledge. Adapting them is difficult. Continuing MLM after adding new vocabulary weakens their contrastive properties. Contrastive-only training barely updates the new token embeddings, because pooling dilutes the gradient signal.

MOSAIC is a multi-stage framework that addresses this conflict. First, a tokenizer trained on in-domain text supplies tokens the base vocabulary lacks. These tokens are added with embeddings initialized as the mean of their existing subwords, and the encoder weights are left unchanged. Next, the model is trained on MLM and contrastive losses together, motivated by a mutual information maximization view. Masking is steered toward domain-relevant tokens so the token-level loss does not overpower the sentence-level one.

The authors tested MOSAIC on a high-resource biomedical domain and a low-resource Islamic domain. It beat strong general-domain baselines by up to 13.4% NDCG@10. The only prerequisite is cheap vocabulary expansion, which makes the method practical for domains with scarce data.

Key points

  • The pipeline has three stages: vocabulary augmentation (Stage1), joint MLM and contrastive optimization (Stage2), and a final Stage3 model. The best reported gain is up to 13.4% NDCG@10 over general-domain baselines.
  • For the Islamic domain, evaluation uses QRCD. Its train and dev splits were merged into 169 test queries, with verified English translations and the Sahih International translation as the retrieval collection.
  • The Islamic setup added 3k domain tokens and initialized from nomic-embed-text-v1 unsup. A loss weight of α = 0.3 gave the most stable and best results.
  • MOSAIC-ID-Stage3 reached 36.8 NDCG@10, beating general-domain models and a contrastive-only Islamic-embed-model. It also had fewer queries with zero NDCG@10.
  • Code and models are released at github.com/rttl-ai/mosaic under Apache-2.0.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Vera Pavlova and Mohammed Makhlouf. 2026. MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.197

@inproceedings{pavlova2026mosaic,
  title = {{MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning}},
  author = {Vera Pavlova and Mohammed Makhlouf},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.197},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.197}
}

References

53 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, Minneapolis, Minnesota, USA. Association for Computati…
  2. Philip Bachman, R. Devon Hjelm, and William Buchwalter. 2019. Learning representations by maximizing mutual information across views. In Neural Information Processing Systems. find
  3. Anas Belfathi, Ygor Gallina, Nicolas Hernandez, Richard Dufour, and Laura Monceaux. 2024. Language model adaptation to specialized domains through selective masking based on genre and topical characteristics. ArXiv, abs/2402.12036. — [arXiv:2402.12036](https://arxiv.org/abs/2402.12036) find
  4. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615– 3620,… find
  5. Rich Caruana. 1997. Multitask learning. Machine Learning, 28(1):41–75. find
  6. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.…
  7. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709.
  8. Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Assoc…
  9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  11. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find
  12. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistic…
  13. John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021. DeCLUTR: Deep contrastive learning for unsupervised textual representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processin…
  14. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020a. Domain-specific language model pretraining for biomedical natural language processing. CoRR, abs/2007.15779.
  15. Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2020b. Train no evil: Selective masking for task-guided pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6966–6974, Online. Association for Computational Lingu… find
  16. R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Adam Trischler, and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. ArXiv, abs/1808.06670. — [arXiv:1808.06670](https://arxiv.org/abs/1808.06670) find
  17. Yongxin Huang, Kexin Wang, Sourav Dutta, Raj Patel, Goran Glavaš, and Iryna Gurevych. 2023. AdaSent: Efficient domain-adapted sentence embeddings for few-shot classification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3420–3434, Singapore. Associ… find
  18. Yining Li, Yuhui Zhang, Xiaoman Pan, Yinan Li, Ning Ding, Wei Wu, Yujing Wang, Xiaoyan Zhu, and Minlie Huang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.
  19. Fangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2021. Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1442–1459, Online…
  20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. RoBERTa: A robustly optimized BERT pretraining approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1–1…
  21. Rana Malhas and Tamer Elsayed. 2020. Ayatec: building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(6):1–21. find
  22. Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul N. Bennett, Jiawei Han, and Xia Song. 2021. Coco-lm: Correcting and contrasting text sequences for language model pretraining. In Neural Information Processing Systems.
  23. Logan Merrick, Cameron Beeler, Mike Lewis, Rohit Girdhar, David Hall, Xilun Chen, John Thickstun, and Jacob Devlin. 2024. Arctic-embed: Scalable, efficient, and accurate text embedding models. arXiv preprint arXiv:2405.05374. — [arXiv:2405.05374](https://arxiv.org/abs/2405.05374) find
  24. Dmitry Kobak, Dong He, et al. 2024. The landscape of biomedical research. bioRxiv. find
  25. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2022. Mteb: Massive text embedding benchmark. In Conference of the European Chapter of the Association for Computational Linguistics.
  26. Lingpeng Kong, Cyprien de Masson d’Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yogatama. 2019. A mutual information maximization perspective of language representation learning. ArXiv, abs/1910.08350. — [arXiv:1910.08350](https://arxiv.org/abs/1910.08350) find
  27. Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. 2024. Nomic embed: Training a reproducible long context text embedder. arXiv preprint arXiv:2402.01613.
  28. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics.
  29. Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119–9130, Online. Association for Computational Linguis…
  30. Vera Pavlova. 2023. Leveraging domain adaptation and data augmentation to improve qur’anic IR in English and Arabic. In Proceedings of ArabicNLP 2023, pages 76–88, Singapore (Hybrid). Association for Computational Linguistics.
  31. Vera Pavlova. 2025. Multi-stage training of bilingual islamic LLM for neural passage retrieval. In Proceedings of the New Horizons in Computational Linguistics for Religious Texts, pages 42–52, Abu Dhabi, UAE. Association for Computational Linguistics. find
  32. Vera Pavlova and Mohammed Makhlouf. 2023. BIOp-timus: Pre-training an optimal biomedical language model with curriculum learning for named entity recognition. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 337–349, Toronto, Canada. Association for Compu… find
  33. Vera Pavlova and Mohammed Makhlouf. 2024. Building an efficient multilingual non-profit IR system for the islamic domain leveraging multiprocessing design in rust. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 981–990, Miami, Florida… find
  34. Vera Pavlova and Mohammed Makhlouf. 2025. Efficient and versatile model for multilingual information retrieval of islamic text: Development and deployment in real-world scenarios. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1239–12… find
  35. Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020. Inexpensive domain adaptation of pretrained language models: Case studies on biomedical NER and covid-19 QA. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1482–1490, Online. Association for Computational Lingu… find
  36. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  37. Vin Sachidananda, Jason Kessler, and Yi-An Lai. 2021. Efficient domain adaptation of language models via adaptive tokenization. In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, pages 155–165, Virtual. Association for Computational Linguistics. find
  38. Nafis Sadeq, Canwen Xu, and Julian McAuley. 2022. InforMask: Unsupervised informative masking for language model pretraining. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5866–5878, Abu Dhabi, United Arab Emirates. Association for Computational Lin… find
  39. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108. find
  40. Tim Schopf, Dennis N. Schneider, and Florian Matthes. 2023. Efficient domain adaptation of sentence embeddings using adapters. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 1046–1053, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. find
  41. Gizem Sogancioglu, Arzucan Ozgur, and Hakime Ozturk. 2017. Biosses: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics, 33(14):i49–i58. find
  42. Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2021. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:…
  43. George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Jens Petersen, Jörg Hakenberg, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition… find
  44. Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748.
  45. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.
  46. Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William Hersh, Kirk Roberts, and Lucy Lu Wang. 2021. Trec-covid: Constructing a pandemic information retrieval test collection. Journal of the American Medical Informatics Association, 28(4):766–773. find
  47. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
  48. Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021. TSDAE: Using transformer-based sequential denoising auto-encoderfor unsupervised sentence embedding learning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 671–688, Punta Cana, Dominican Republic. Association for C… find
  49. Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022a. GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
  50. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022b. Text embeddings by weakly-supervised contrastive pre-training. ArXiv, abs/2212.03533.
  51. Xing Wu, Chaochen Gao, Zijia Lin, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2022. InfoCSE: Information-aggregated contrastive learning of sentence embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3060–3070, Abu Dhabi, United Arab Emirates. Association fo… find
  52. Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466. find
  53. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian yun Nie. 2023. C-pack: Packed resources for general chinese embeddings. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pavlova-2026-mosaic-masked-objective-selective-adaptation/