Elamur Research⌕
☰
home / papers / 2024 / Dreaming Out Loud: A Self-Synthesis Approach For Training Vision-Language Models With Developmentally Plausible Data

Dreaming Out Loud: A Self-Synthesis Approach For Training Vision-Language Models With Developmentally Plausible Data

Summary

Modern large language models are trained on trillions of tokens. A human, by contrast, hears at most about 100M words before adolescence. The authors worked within the BabyLM Challenge and investigated whether a multimodal model can be trained on such a limited budget. To this end, they assembled their own 100M-word corpus: half consists of texts from FineWeb-Edu selected for educational quality, and the rest are image–text pairs. The BabyLLaMA model was trained in four stages, by analogy with child development. First, it acquires language from text alone. Next, it learns to describe images through an attached DINOv2 encoder. In the third stage, it generates captions for unlabeled images itself and is further trained on a mixture of these synthetic texts and real ones, which resembles dreaming. Finally, it is fine-tuned on questions and visual reasoning. Each stage yielded a small gain specifically on its own type of task. Adding synthetic text improved the results on language benchmarks, or at least did not worsen them. The work shows that a model can partly create its own training data when real texts are scarce.

Key points

  • Data (100M words): 50M — text from FineWeb-Edu; 35M — image captions from WIT (27M), OBELICS (5M) and LAION (3M); 15M — the M3IT multimodal instruction dataset. For comparison: LLaMA-3-8B was trained on a volume roughly 150,000 times larger.
  • Model: BabyLLaMA has 58M parameters (16 Transformer layers, hidden size 512, 8 attention heads) and a BPE tokenizer with 16,000 tokens. The DINOv2 Large encoder is frozen and outputs 256 visual tokens, which a two-layer MLP projector passes to the language model.
  • Self-synthesis (stage 3): the model was further trained for 2 epochs on 85M real words and 42M words it generated itself.
  • Results: the model after stage 3 performs best on BLiMP and EWoK, and after stage 4 on GLUE, VQA and Winoground. On BLiMP Supplement, the highest score belongs to the stage 2 model. Language benchmarks, except GLUE (fine-tuned via LoRA), were evaluated without fine-tuning (zero-shot).
  • Ablation (with / without synthetic text): BLiMP — 0.736 / 0.736, BLiMP Supplement — 0.556 / 0.550, EWoK — 0.514 / 0.510. The authors themselves acknowledge that the gain is modest, and propose repeating the stages cyclically or applying layer-fusion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2411.00828

@inproceedings{alkhamissi2024dreaming,
  title = {{Dreaming Out Loud: A Self-Synthesis Approach For Training Vision-Language Models With Developmentally Plausible Data}},
  author = {AlKhamissi, Badr and Tang, Yingtian and Gökce, Abdülkadir and Mehrer, Johannes and Schrimpf, Martin},
  booktitle = {The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning},
  year = {2024},
  eprint = {2411.00828},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.00828}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457. find
  2. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, ab… find
  3. Jill Gilkerson, Jeffrey A. Richards, Steven F. Warren, Judith K. Montgomery, Charles R. Greenwood, D. Kim-brough Oller, John H. L. Hansen, and Terrance D. Paul. 2017. Mapping the early language environment using all-day recordings and automated analysis. American Journal of Speech-Language Pathology… find
  4. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, 127:398 – 414. find our card
  5. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR). find our card
  6. Erik Hoel. 2021. The overfitted brain: Dreams evolved to assist generalization. Patterns, 2(5):100244. find
  7. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. find
  8. Anna Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H Clark, Carina Kauf, Jennifer Hu, Pramod RT, Gabriel Grand, Vivian Paulun, Maria Ryskina, Ekin Akyurek, Ethan Wilcox, Nafisa Rashid, Leshem Choshen, Roger Levy, Evelina Fedorenko, Josh Tenenbaum, and Jacob Andreas.… find
  9. Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. 2024. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems,… find our card
  10. Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. 2023. M 3 it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306.04387. — [arXiv:2306.04387](https://arxiv.org/abs/2306.04387) find
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, and et al. 2024. The llama 3 herd of models. ArXiv, abs/2407.21783. find
  12. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. Fineweb-edu. find
  13. Muhammad ElNokrashy, Badr AlKhamissi, and Mona Diab. 2024. Depth-wise attention (DWAtt): A layer fusion method for data-efficient classification. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4665–… find
  14. Gary F. Marcus. 2020. The next decade in ai: Four steps towards robust artificial intelligence. ArXiv, abs/2002.06177. find
  15. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association fo… find
  16. Maxime Oquab, Timoth’ee Darcet, Théo Moutakanni, Huy Q. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russ Howes, Po-Yao (Bernie) Huang, Shang-Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sh… find our card
  17. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. Preprint, arXiv:2406.17557. find our card
  18. Luke Y. Prince and Blake A. Richards. 2021. The overfitted brain hypothesis. Patterns, 2(5):100268. find
  19. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114. of the BabyLM Challenge at the 27th… find our card
  20. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mos-quera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell, editors. 2023b. Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning. Associat… find
  21. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2019. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377–392. find
  22. Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2015. Yin and yang: Balancing and answering binary visual questions. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5014–5022. find
  23. Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. 2021. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 24… find
  24. Alvin Wei Ming Tan, Sunny Yu, Bria Long, Wanjing Anya Ma, Tonya Murray, Rebecca D. Silverman, Jason D. Yeatman, and Michael C. Frank. 2024. Devbench: A multimodal developmental benchmark for language learning. ArXiv, abs/2406.10215. find
  25. Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR. find
  26. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… find our card
  27. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mos-quera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023a. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/alkhamissi-2024-dreaming-out-loud-self-synthesis/