Elamur Research⌕
☰
home / papers / 2026 / Early Fusion with Contrastive Learning: A Lightweight Alternative for Multi-modal Classification

Early Fusion with Contrastive Learning: A Lightweight Alternative for Multi-modal Classification

Summary

Leading multimodal classifiers usually depend on large, computationally expensive architectures. This paper asks whether a much smaller design can reach similar accuracy. The authors build a framework around early fusion. Pretrained encoders turn each modality into an embedding: CLIP for text, images and video frames, AutoGluon-Tabular for tables, CodeBERT for code, and Wav2Vec for audio. These embeddings are concatenated and passed to a compact feed-forward classifier. Because text and image tend to be the strongest signals, weaker modalities first go through a residual projector. That projector is trained with a contrastive objective that pulls each modality's embedding toward the matching text and image embeddings, using hard negative mining. Tests on three kinds of multimodal classification datasets show the lightweight model matching, and sometimes beating, much larger state-of-the-art systems. Per-dataset scores were in a part of the paper not available for this summary. The work matters because it gives an accessible baseline for settings where compute or budget is limited.

Key points

  • Model size: the early-fusion model has about 1M trainable parameters, compared with 355M for the AutoGluon baseline.
  • Classifier: two blocks, each made of a fully connected layer, batch normalization and ReLU, trained with cross-entropy loss. Video is represented by 15 frames encoded with CLIP and then max-pooled.
  • Projector: a linear layer with Tanh activation, followed by dropout and a residual connection back to the input features.
  • Alignment loss: a modified NT-Xent loss built on temperature-scaled cosine similarity. Only the top-k hardest negatives per sample are used, which reduces noise.
  • Code: released for reproducibility at github.com/fWern/MultimodelClassification.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3ioe7epfs78m

@inproceedings{wernlein2026early,
  title = {{Early Fusion with Contrastive Learning: A Lightweight Alternative for Multi-modal Classification}},
  author = {Wernlein, Felix and Jana, Abhik and Sikdar, Sandipan},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3ioe7epfs78m},
  url = {https://doi.org/10.63317/3ioe7epfs78m}
}

References

36 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Haojie Chen, Zhuo Wang, Hongde Qin, and Xiaokai Mu. 2024. Dhfnet: Decoupled hierarchical fusion network for rgb-t dense prediction tasks. Neurocomputing, 583:127594. find
  2. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR. find our card
  3. Ahmet Demirkaya, Jiasi Chen, and Samet Oymak. 2020. Exploring the role of loss functions in multiclass classification. In 2020 54th annual conference on information sciences and systems (ciss), pages 1–5. IEEE. find
  4. Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. 2020. Autogluon-tabular: Robust and accurate automl for structured data. arXiv preprint arXiv:2003.06505. — [arXiv:2003.06505](https://arxiv.org/abs/2003.06505) find
  5. Nick Erickson, Xingjian Shi, James Sharpnack, and Alexander Smola. 2022. Multimodal automl for image, text and tabular data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4786– 4787. find
  6. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155. find our card
  7. Konrad Gadzicki, Razieh Khamsehashari, and Christoph Zetzsche. 2020a. Early vs late fusion in multimodal convolutional neural networks. In 2020 IEEE 23rd international conference on information fusion (FUSION), pages 1–6. IEEE. find
  8. Konrad Gadzicki, Razieh Khamsehashari, and Christoph Zetzsche. 2020b. Early vs late fusion in multimodal convolutional neural networks. In 2020 IEEE 23rd International Conference on Information Fusion (FUSION), pages 1–6. find
  9. Valerio Guarrasi, Fatih Aksu, Camillo Maria Caruso, Francesco Di Feola, Aurora Rofena, Filippo Ruffini, and Paolo Soda. 2024. A systematic review of intermediate fusion in multimodal deep learning for biomedical applications. arXiv preprint arXiv:2408.02686. — [arXiv:2408.02686](https://arxiv.org/abs/2408.02686) find
  10. Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM international conference on multimedia, pages 1122–1131. find
  11. Douwe Kiela, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2018. Efficient large-scale multimodal classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32. find
  12. Jin-Hwa Kim, Sang-Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. 2016. Multimodal residual learning for visual qa. Advances in neural information processing systems, 29. find
  13. Changwon Kwak, Pilsu Jung, and Seonah Lee. 2023. A multimodal deep learning model using text, image, and code data for improving issue classification tasks. Applied Sciences, 13(16):9456. find
  14. Ronghao Lin and Haifeng Hu. 2023. Missmodal: Increasing robustness to missing modality in multimodal sentiment analysis. Transactions of the Association for Computational Linguistics, 11:1686– 1702. find
  15. Y Liu. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  16. Serveh Lotfi, Mitra Mirzarezaee, Mehdi Hosseinzadeh, and Vahid Seydi. 2021. Detection of rumor conversations in twitter using graph convolutional networks. Applied Intelligence, 51:4774– 4787. find
  17. Jiaying Lu, Yongchen Qian, Shifan Zhao, Yuanzhe Xi, and Carl Yang. 2023. Mug: A multimodal classification benchmark on game data with tabular, textual, and visual fields. arXiv preprint arXiv:2302.02978. find
  18. Krishna D N. 2021. Using large pre-trained models with cross-modal attention for multi-modal emotion recognition. find
  19. Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. 2011. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 689–696. find
  20. Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, Ben Glocker, and Daniel Rueckert. 2018. Attention u-net: Learning where to look for the pancreas. find
  21. Shengsheng Qian, Jun Hu, Quan Fang, and Changsheng Xu. 2021. Knowledge-aware multi-modal adaptive graph convolutional networks for fake news detection. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 17(3):1–23. find
  22. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PM… find our card
  23. Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque. 2020. Integrating multimodal information in large pretrained transformers. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2020, page 2359. NI… find our card
  24. Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. In Proc. find
  25. Interspeech 2019, pages 3465–3469. find
  26. Xingjian Shi, Jonas Mueller, Nick Erickson, Mu Li, and Alex Smola. 2021. Multimodal automl on structured tables with text fields. In 8th ICML Workshop on Automated Machine Learning (AutoML). find
  27. Cees G. M. Snoek, Marcel Worring, and Arnold W. M. Smeulders. 2005. Early versus late fusion in semantic video analysis. In Proceedings of the 13th Annual ACM International Conference on Multimedia, MULTIMEDIA ’05, page 399–402, New York, NY, USA. Association for Computing Machinery. find
  28. Sören Richard Stahlschmidt, Benjamin Ulfenborg, and Jane Synnergren. 2022. Multimodal deep learning for biomedical data fusion: a review. Briefings in Bioinformatics, 23(2):bbab569. find
  29. Ashish Vaswani. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. find
  30. Weiyao Wang, Du Tran, and Matt Feiszli. 2020. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12695–12705. find
  31. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250. find our card
  32. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259. — [arXiv:1606.06259](https://arxiv.org/abs/1606.06259) find
  33. AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume… find our card
  34. Jin Zeng, Yanfeng Tong, Yunmu Huang, Qiong Yan, Wenxiu Sun, Jing Chen, and Yongtian Wang. 2019. Deep surface normal estimation with hierarchical rgb-d fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6153–6162. find
  35. Fei Zhao, Chengcui Zhang, and Baocheng Geng. 2024. Deep multimodal data fusion. ACM Computing Surveys, 56(9):1–36. find
  36. Qiankun Zhao, Yingcai Wan, Jiqian Xu, and Lijin Fang. 2023. Cross-modal attention fusion network for rgb-d semantic segmentation. Neurocomputing, 548:126389. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wernlein-2026-early-fusion-contrastive-learning-lightweight/