Elamur Research⌕
☰
home / papers / 2026 / Mind Your Special Tokens! On the Importance of Dedicated Sequence-End Tokens in Vision-Language Embedding Models

Mind Your Special Tokens! On the Importance of Dedicated Sequence-End Tokens in Vision-Language Embedding Models

Paper · 2026 · Elio Musacchio, Giovanni Semeraro and Goran Glavaš

Summary

Recent work turns generative Large Vision-Language Models (LVLMs) into vision-language encoders (VLEs) through contrastive fine-tuning. These VLEs usually take the output state of whatever token happens to come last as the sequence embedding, with no dedicated end marker. The authors ask whether this habit makes the models fragile. They first test three public VLEs on MMEB, giving them the same instances in three versions that differ only in how the instruction ends: punctuation plus newline, punctuation only, or neither. They then train their own encoders from Qwen2.5-VL-3B-Instruct, varying three things:

  • the pooling method (mean pooling with bidirectional attention, or last-token pooling),
  • whether an explicit <EOS> token is appended,
  • whether trailing newlines are stripped from the training data.

Last-token pooling without <EOS> turns out to be very sensitive to these trivial endings, with large drops on VQA and Retrieval. Adding <EOS>, or switching to bidirectional mean pooling, largely removes the problem. Mean pooling is the most stable option, which challenges the current default design of LLM-based embedding models.

Key points

  • Off-the-shelf models: VLM2Vec-Qwen2VL-2B, VLM2Vec-Qwen2VL-7B and mmE5-mllama-11B-instruct all perform best when inputs end with punctuation and a newline. Removing these characters can cost up to about 30 P@1 points on VQA and Retrieval.
  • Benchmark: evaluation uses MMEB, which has 37 tasks in four groups: Classification, VQA, Retrieval and Visual Grounding.
  • Own models, last-token pooling without <EOS> (No-Strip training): on PLAIN inputs, VQA falls from 54.2 to 34.7 and Retrieval from 61.2 to 41.1.
    • With <EOS>, the PU+NL vs. PLAIN gap shrinks to 4.9 (VQA) and 2.8 (Retrieval).
    • With mean pooling, the gap is only 0.9 and 0.7.
  • Ablations:
    • Mean pooling performs clearly worse when the causal mask is kept, especially on VQA.
    • Training with randomly mixed input endings is less effective than adding <EOS>.
    • Stripping newlines from training data helps, but less than either of the main fixes.
  • Release and limitations: code and models are available at github.com/m-elio/LVLMs-Encoder. The study covers English only and models of 2B–11B parameters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PisaPisa · Italy25 works · 3,842 citations🎓University of Bari Aldo MoroBari · Italy6 works · 2,210 citations🎓University of WürzburgWürzburg · Germany8 works · 60 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Elio Musacchio, Giovanni Semeraro and Goran Glavaš. 2026. Mind Your Special Tokens! On the Importance of Dedicated Sequence-End Tokens in Vision-Language Embedding Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-short.24

@inproceedings{musacchio2026mind,
  title = {{Mind Your Special Tokens! On the Importance of Dedicated Sequence-End Tokens in Vision-Language Embedding Models}},
  author = {Elio Musacchio, Giovanni Semeraro and Goran Glavaš},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-short.24},
  url = {https://doi.org/10.18653/v1/2026.eacl-short.24}
}

References

23 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this project by providing computing time on the GCS Supercomputer JURECA at Jülich Supercomputing Centre (JSC) (Jülich Supercomputing Centre, 2021). We acknowledge the support of the PNRR pr…
  2. Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. 2025. mme5: Improving multimodal multilingual embeddings via high-quality synthetic data. Preprint, arXiv:2502.08468. find
  3. Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. 2021. Scaling deep contrastive learning batch size under memory limited setup. In Proceedings of the 6th Workshop on Representation Learning for NLP, RepL4NLP@ACL-IJCNLP 2021, Online, August 6, 2021, pages 316–321. Association for Computational Li…
  4. Gregor Geigle, Florian Schneider, Carolin Holtermann, Chris Biemann, Radu Timofte, Anne Lauscher, and Goran Glavaš. 2025. Centurio: On drivers of multilingual ability of large vision-language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1:…
  5. Weiquan Huang, Aoqi Wu, Yifan Yang, Xufang Luo, Yuqing Yang, Liang Hu, Qi Dai, Xiyang Dai, Dongdong Chen, Chong Luo, and Lili Qiu. 2024. LLM2CLIP: powerful language model unlocks richer visual representation. CoRR, abs/2411.04997. — [arXiv:2411.04997](https://arxiv.org/abs/2411.04997) find
  6. Yongxin Huang, Kexin Wang, Goran Glavaš, and Iryna Gurevych. 2025. Modular sentence encoders: Separating language specialization from cross-lingual alignment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2167–2187, Vienna,… find
  7. Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. 2025. Vlm2vec: Training vision-language models for massive multimodal embedding tasks. In ICLR. find
  8. Jülich Supercomputing Centre. 2021. JURECA: Data Centric and Booster Modules implementing the Modular Supercomputing Architecture at Jülich Supercomputing Centre. Journal of large-scale research facilities, 7:A182. find
  9. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  10. Llama Team. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783.
  11. Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, Yingbo Zhou, Wenhu Chen, and Semih Yavuz. 2025. Vlm2vec-v2: Advancing multimodal embedding for videos, images, and visual documents. CoRR, abs/2507.04590. — [arXiv:2507.04590](https://arxiv.org/abs/2507.04590) find
  12. Niklas Muennighoff. 2022. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904.
  13. Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, and 1 others. 2022. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10005.
  14. Lilian Ngweta, Kiran Kate, Jason Tsay, and Yara Rizk. 2025. Towards LLMs robustness to changes in prompt format styles. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Resear… find
  15. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8…
  16. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, Ch…
  17. Fabian David Schmidt, Philipp Borchert, Ivan Vulić, and Goran Glavaš. 2024. Self-distillation for model stacking unlocks cross-lingual NLU in 200+ languages. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6724–6743, Miami, Florida, USA. Association for Computational… find
  18. Weijie Tu, Weijian Deng, and Tom Gedeon. 2023. A closer look at the robustness of contrastive language-image pre-training (CLIP). In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10… find
  19. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024a. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916, Bangkok, Thailand. Ass…
  20. Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. 2024b. A sober look at the robustness of clips to spurious features. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, C… find
  21. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory e…
  22. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2526–2547, Vienna, Austria. Association for Computational Linguistics. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Li… find
  23. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/musacchio-2026-mind-your-special-tokens-importance/