Elamur Research⌕
☰
home / papers / 2025 / BitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices

BitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices

Summary

The authors address the problem of running multimodal (text + image) language models on edge devices, which have little memory and limited compute. Large full-precision VLMs such as BLIP-2 or Flamingo are unsuitable there. To address this, they propose BitMar, a very small model with 14M parameters. A BitNet-style text encoder and a compression module for image features from a frozen DiNOv2 operate with ternary (1.58-bit) weights. The two modalities are fused via cross-attention, and this joint representation serves as a query to an external episodic memory with a fixed number of slots. The vector retrieved from memory is fed into every decoder layer. To process long token streams with a constant memory footprint, the decoder combines attention sinks with a sliding window. The model was trained for BabyLM on 100M tokens: half are image captions from CC3M and Localized Narratives, and half are the BabyLM text corpus. On knowledge-intensive tasks, BitMar falls noticeably behind larger low-bit models, but it is competitive on BoolQ and WinoGrande. An ablation shows that episodic memory sharply speeds up inference and reduces energy consumption. Thus, aggressive quantization combined with memory can yield a viable model for edge devices.

Key points

  • Architecture: the encoder and the decoder each have 4 layers (d=128, 4 heads, up to 256 tokens). DiNOv2 features are compressed from 768 to 128 dimensions by a two-layer MLP. The episodic memory is a 512×128 matrix. The GPT-2 tokenizer is used. The loss function combines the LM loss, InfoNCE with a weight of 1.5, and a memory stability regularizer with a weight of 0.1.
  • Comparison with low-bit models (Table 1): BoolQ 42.8, WinoGrande 54.6, ARC-Easy 28.3, HellaSwag 30.0, CommonsenseQA 24.6, MMLU 27.9. By comparison, BitNet b1.58 2B scores considerably higher on most of these tasks.
  • BabyLM evaluation: the average result on the finetune tasks is 60.5% (QQP 70.2%, MRPC 69.1%, BoolQ 66.5%). The multimodal tasks yield only 21–25%, the best being EWoK (24.9%). BLiMP is 48.7%. The correlations on WUG are negative, meaning that morphological productivity is weak.
  • Memory ablation: with and without memory, the model achieves 57.3 and 7.7 tok/s, a per-token latency of 17.3 and 129.8 ms, energy of 1.90 and 9.17 J, and RAM of 956 and 1,076 MB, respectively. Memory adds roughly 3–4 pp on entity tracking and VQA. At the same time, it worsens WUG and the psycholinguistic alignment metrics for reading.
  • Compression ratio: the share of zero ternary weights grows during training and stabilizes at 42.8%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.babylm-main.11 · arXiv 2510.10560

@inproceedings{aman2025bitmar,
  title = {{BitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices}},
  author = {Aman, Euhid and Carlin, Esteban and Pao, Hsing-Kuo Kenneth and Beltrame, Giovanni and Sari, Ghaluh Indah Permata and Chen, Yie-Tarng},
  booktitle = {Proceedings of the First BabyLM Workshop},
  year = {2025},
  doi = {10.18653/v1/2025.babylm-main.11},
  eprint = {2510.10560},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.babylm-main.11}
}

References

28 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millicah, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock… find our card
  2. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering . In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6077–6086, Los Alamit… find
  3. Antoine Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Sebastian Rutherford, Matthew Botvinick, Jean-Baptiste Sifre, and Stan Clark. 2022. Improving language models by retrieving from trillions of tokens. International Conference on Machine Learning (ICML), 162:2209–2226. find
  4. Lucas Charpentier, Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Michael Hu, Jaap Jumelet, Tal Linzen, Jing Liu, Aaron Mueller, Candace Ross, Raj Sanjay Shah, Alex Warstadt, Ethan Wilcox, and Adina Williams. 2025. BabyLM turns 3: Call for papers for the 2025 BabyLM workshop. In BabyLM Turns 3: C… find
  5. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. CoRR, abs/1504.00325. find our card
  6. Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. 2016. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471–476. find
  7. Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni… find
  8. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. find our card
  9. Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings… find our card
  10. Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. 2024. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. arXiv preprint arXiv:2402.14905. find
  11. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Curran Associates Inc., Red Hook, NY, USA. find our card
  12. Marcelo G. Mattar and Nathaniel D. Daw. 2022. A neural network model of when to retrieve and encode episodic memories. eLife, 11:e74445. find
  13. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2021. 8-bit optimizers via block-wise quantization. CoRR, abs/2110.02861. find our card
  14. Marcelo G. Mattar and Nathaniel D. Daw. 2024. Toward the emergence of intelligent control: Episodic generalization and optimization. Open Mind. find
  15. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: accurate post-training quantization for generative pre-trained transformers. CoRR, abs/2210.17323. find
  16. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation Learning with Contrastive Predictive Coding. arXiv:1807.03748 [cs, stat]. ArXiv: 1807.03748. find our card
  17. Shaikat Galib, Shanshan Wang, Guanshuo Xu, Pascal Pfeiffer, Ryan Chesler, Mark Landry, and Sri Satish Ambati. 2024. H2ovl-mississippi vision language models technical report. find
  18. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel… find our card
  19. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and… find our card
  20. Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. 2020. Connecting vision and language with localized narratives. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V, page 647–664, Berlin, Heidelberg.… find
  21. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565,… find our card
  22. Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. 2019. And the bit goes down: Revisiting the quantization of neural networks. CoRR, abs/1907.05686. — [arXiv:1907.05686](https://arxiv.org/abs/1907.05686) find
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  24. Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. 2023. Bitnet: Scaling 1-bit transformers for large language models. find
  25. Yuanhan Yao, Qinghao Yu, Ao Zhang, Xiaoyi Wang, Zhiyang Xu, Chendong Yuan, Ying Wang, Yaoyao Liu, Kunchang Wang, Yunhai Yu, et al. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800. find our card
  26. Zhengqing Yuan, Zhaoxu Ren, Lichao Feng, Zhi Zhao, Kai Cui, and Shiliang Jiang. 2024. Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint arXiv:2312.16862. — [arXiv:2312.16862](https://arxiv.org/abs/2312.16862) find
  27. Wei Zhang, Xiaoming Liu, Hao Chen, and Yifan Wang. 2024. Tinyllm: A framework for training and deploying language models at the edge computers. arXiv preprint arXiv:2412.15304. find
  28. Chenzhuo Zhu, Song Han, Huizi Mao, and William J. Dally. 2016. Trained ternary quantization. CoRR, abs/1612.01064. — [arXiv:1612.01064](https://arxiv.org/abs/1612.01064) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aman-2025-bitmar-low-bit-multimodal-fusion/