Elamur Research⌕
☰
home / papers / 2024 / DevBench: A multimodal developmental benchmark for language learning

DevBench: A multimodal developmental benchmark for language learning

Paper · 2024 · Alvin Wei Ming Tan, Sunny Yu, Bria Long et al.

Summary

Vision–language models need far more linguistic input than children do. Models trained on smaller or child-like naturalistic data are usually judged by adult-oriented benchmarks that probe few language skills and rarely include human comparison data. The authors address this with DevBench, a multimodal suite of seven tasks covering lexicon, syntax and semantics. Each task comes with item-level responses from children and adults. In the lexical and syntactic tasks, the model selects the picture that fits a spoken word or sentence, which mirrors how young children are tested without speaking. The semantic tasks compare representational similarity structures. Scoring rewards resemblance to human response distributions rather than raw accuracy, using optimised KL divergence as the linking measure. The authors tested several vision–language models, including small and developmentally trained ones, and found that human-likeness varied widely. Models that were more accurate on a task usually also matched humans more closely, while model size mattered less. OpenCLIP-H checkpoints moved toward adult patterns during training on some tasks but not others. DevBench therefore offers a child-comparable yardstick for models of language acquisition.

Key points

  • The seven tasks span three domains. Lexical tasks include Looking-while-listening (LWL) and the visual vocabulary task (VV). Syntax is tested with the Test of Receptive Grammar (TROG). Semantic tasks are based on representational similarity.
  • LWL combines three child datasets at 18, 24 and 30 months (Adams et al., Frank et al., Donnelly & Kidd) with a total N of 294. VV draws on unpublished data from Long et al. covering 1,780 participants aged 3–12 plus adults.
  • Model–human similarity correlated strongly with task accuracy and more weakly with model size. Models also handled ambiguous inputs worse than humans did.
  • The authors note several limitations: the data are mostly English, some tasks have few items or participants, and there is a domain gap between the cartoon stimuli used in TROG and CLIP's photographic training data.
  • Code and data are released at github.com/alvinwmtan/dev-bench under CC BY-NC-SA or more permissive licences. The paper was presented in the NeurIPS 2024 Datasets and Benchmarks track.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksVision & language · 657 works

Cite

Cite as Alvin Wei Ming Tan, Sunny Yu, Bria Long, Wanjing Anya Ma, Tonya Murray, Rebecca D. Silverman, Jason D. Yeatman and Michael C. Frank. 2024. DevBench: A multimodal developmental benchmark for language learning. arXiv:2406.10215.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.10215 · arXiv 2406.10215

@misc{tan2024devbench,
  title = {{DevBench: A multimodal developmental benchmark for language learning}},
  author = {Alvin Wei Ming Tan, Sunny Yu, Bria Long, Wanjing Anya Ma, Tonya Murray, Rebecca D. Silverman, Jason D. Yeatman and Michael C. Frank},
  year = {2024},
  doi = {10.48550/arxiv.2406.10215},
  eprint = {2406.10215},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.10215}
}

References

72 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael C. Frank. Bridging the data gap between children and large language models. Trends in Cognitive Sciences, 27(11):990–992, 2023. ISSN 1364-6613. doi: 10.1016/j.tics.2023.08.007. — [doi:10.1016/j.tics.2023.08.007](https://doi.org/10.1016/j.tics.2023.08.007) find
  2. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora. In Alex Warstadt, Aaron Mue… find
  3. Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language. In Arianna Bisazza and Omri Abend, editors, Proceedings of the 25th Conference on Computational Natural Language Learning, pages 624–646. Association for Computati…
  4. Wai Keen Vong, Wentao Wang, A. Emin Orhan, and Brenden M. Lake. Grounded language acquisition through the eyes and ears of a single child. Science, 383(6682):504–511, 2024. doi: 10.1126/science. adi1374. — [doi:10.1126/science](https://doi.org/10.1126/science) find
  5. Wentao Wang, Wai Keen Vong, Najoung Kim, and Brenden M. Lake. Finding Structure in One Child’s Linguistic Experience. Cognitive Science, 47(6):e13305, 2023. ISSN 1551-6709. doi: 10.1111/cogs.13305. — [doi:10.1111/cogs.13305](https://doi.org/10.1111/cogs.13305) find
  6. Arielle Borovsky. Lexico-semantic structure in vocabulary and its links to lexical processing in toddlerhood and language outcomes at age three. Developmental Psychology, 58(4):607–630, 2022. ISSN 1939-0599. doi: 10.1037/dev0001291. — [doi:10.1037/dev0001291](https://doi.org/10.1037/dev0001291) find
  7. Ellen Irén Brinchmann, Johan Braeken, and Solveig-Alma Halaas Lyster. Is there a direct relation between the development of vocabulary and grammar? Developmental Science, 22(1):e12709, 2019. ISSN 1467-7687. doi: 10.1111/desc.12709. — [doi:10.1111/desc.12709](https://doi.org/10.1111/desc.12709) find
  8. Laura Justice, Kate Cain, Hui Jiang, Jessica Logan, Rongfang Jia, Hui Jiang, Jessica A. Logan, and Rongfang Jia. Modeling the Nature of Grammar and Vocabulary Trajectories From Prekindergarten to Third Grade. Journal of Speech, Language, and Hearing Research, 61(4):910–923, 2018. doi: 10.1044/2018_J… — [doi:10.1044/2018_jslhr-l-17-0090](https://doi.org/10.1044/2018_jslhr-l-17-0090) find
  9. Margaux Keith and Elena Nicoladis. The role of within-language vocabulary size in children’s semantic development: Evidence from bilingual children. Journal of Child Language, 40(4):873–884, 2013. ISSN 0305-0009, 1469-7602. doi: 10.1017/S0305000912000268. — [doi:10.1017/s0305000912000268](https://doi.org/10.1017/s0305000912000268) find
  10. Jessie Ricketts, Nicola Dawson, and Robert Davies. The hidden depths of new word knowledge: Using graded measures of orthographic and semantic learning to measure vocabulary acquisition. Learning and Instruction, 74:101468, 2021. ISSN 0959-4752. doi: 10.1016/j.learninstruc.2021.101468. — [doi:10.1016/j.learninstruc.2021.101468](https://doi.org/10.1016/j.learninstruc.2021.101468) find
  11. Michael C Frank. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, 2(8):451–452, 2023. find
  12. Anna A. Ivanova. Running cognitive evaluations on large language models: The do’s and the don’ts, 2023. find
  13. Jennifer Hu and Michael C Frank. Auxiliary task demands mask the capabilities of smaller language models. arXiv preprint arXiv:2404.02418, 2024.
  14. Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible Scaling Laws for Contrastive Language-Image Learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page…
  15. Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):423–443, 2019. doi: 10.1109/TPAMI.2018.2798607. — [doi:10.1109/tpami.2018.2798607](https://doi.org/10.1109/tpami.2018.2798607) find
  16. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, 2016. URL https://arxiv.org/abs/1612.06890. ↗
  17. Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulić. IGLUE: A benchmark for transfer learning across modalities, tasks, and languages. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, edito… find
  18. Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, Jing Shao, and Wanli Ouyang. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark, 2023. find
  19. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
  20. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024.
  21. Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, joseph heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alexandre Fréchette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Wi… find
  22. Lin Su, Nan Duan, Edward Cui, Lei Ji, Chenfei Wu, Huaishao Luo, Yongfei Liu, Ming Zhong, Taroon Bharti, and Arun Sacheti. GEM: A general evaluation benchmark for multimodal tasks. CoRR, abs/2106.09889, 2021. URL https://arxiv.org/abs/2106.09889. ↗
  23. Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A. Lee, Yuke Zhu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Multibench: Multiscale benchmarks for multimodal representation learning. CoRR, abs/2107.07502, 2021. URL https://arxiv.org/abs/ 21… ↗
  24. Eliza Kosoy, Emily Rose Reagan, Leslie Lai, Alison Gopnik, and Danielle Krettek Cobb. Comparing Machines and Children: Using Developmental Psychology Experiments to Assess the Strengths and Weaknesses of LaMDA Responses. In Proceedings of the First Workshop on AI Meets Moral Philosophy and Moral Psy… — [doi:10.48550/arxiv.2305.11243](https://doi.org/10.48550/arxiv.2305.11243) find
  25. Luca Weihs, Amanda Rose Yuile, Renée Baillargeon, Cynthia Fisher, Gary Marcus, Roozbeh Mottaghi, and Aniruddha Kembhavi. Benchmarking progress to infant-level physical reasoning in ai. TMLR, 2022. find
  26. Guangyuan Jiang, Manjie Xu, Shiji Xin, Wei Liang, Yujia Peng, Chi Zhang, and Yixin Zhu. MEWL: Few-shot multimodal word learning with referential uncertainty. In Proceedings of the 40th International Conference on Machine Learning, pages 15144–15169. PMLR, 2023. URL https://proceedings. mlr.press/v20… find
  27. Saber Sheybani, Linda B. Smith, Zoran Tiganj, Sahaj Singh Maini, and Aravind Dendukuri. ModelVsBaby: A Developmentally Motivated Benchmark of Out-of-Distribution Object Recognition, 2024. find
  28. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension, 2023.
  29. Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems,… find
  30. Tyler A. Chang and Benjamin K. Bergen. Word Acquisition in Neural Language Models. Transactions of the Association for Computational Linguistics, 10:1–16, 2022. ISSN 2307-387X. doi: 10.1162/tacl_a_00444. find
  31. Linnea Evanson, Yair Lakretz, and Jean Rémi King. Language acquisition: Do children and language models follow similar learning stages? In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 12205–12218. Association…
  32. Anne E. Fernald, Renate Zangl, Ana Luz Portillo, and Virginia A. Marchman. Using Eye Movements to Monitor Spoken Language Comprehension by Infants and Young Children: Looking While Listening. Language Acquisition and Language Disorders. John Benjamins Publishing Company, 2008. ISBN 978-90-272-5304-0… — [doi:10.1075/lald.44.06fer](https://doi.org/10.1075/lald.44.06fer) find
  33. Katherine A. Adams, Virginia A. Marchman, Elizabeth C. Loi, Melanie D. Ashland, Anne Fernald, and Heidi M. Feldman. Caregiver Talk and Medical Risk as Predictors of Language Outcomes in Full Term and Preterm Toddlers. Child Development, 89(5):1674–1690, 2018. ISSN 1467-8624. doi: 10.1111/cdev.12818. — [doi:10.1111/cdev.12818](https://doi.org/10.1111/cdev.12818) find
  34. Michael C. Frank, Elise Sugarman, Alexandra C. Horowitz, Molly L. Lewis, and Daniel Yurovsky. Using Tablets to Collect Data From Young Children. Journal of Cognition and Development, 17(1):1–17, 2016. ISSN 1524-8372. doi: 10.1080/15248372.2015.1061528. — [doi:10.1080/15248372.2015.1061528](https://doi.org/10.1080/15248372.2015.1061528) find
  35. Seamus Donnelly and Evan Kidd. Onset Neighborhood Density Slows Lexical Access in High Vocabulary 30-Month Olds. Cognitive Science, 45(9):e13022, 2021. ISSN 1551-6709. doi: 10.1111/cogs.13022. — [doi:10.1111/cogs.13022](https://doi.org/10.1111/cogs.13022) find
  36. Bria Long, Wanjing Anya Ma, Rebecca D. Silverman, Jason D. Yeatman, and Michael C. Frank. Developmental changes in the precision of visual concept knowledge. In Annual Meeting of the Vision Sciences Society, 2024. find
  37. Dorothy V. M. Bishop. Test for the Reception of Grammar (TROG). Medical Research Council, 2nd ed edition, 1989. find
  38. Rebecca D. Silverman and Jason D. Yeatman. Accelerating literacy with digital technology, 2023. find
  39. Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5228–5238, 2022. doi: 10… find
  40. Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, and Kyle Mahowald. Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process… find
  41. Doris R. Entwisle. Word Associations of Young Children / by Doris R. Entwisle. Johns Hopkins Press, 1966. find
  42. Douglas L. Nelson, Cathy L. McEvoy, and Thomas A. Schreiber. The University of South Florida word association, rhyme, and word fragment norms., 1998. URL http://w3.usf.edu/FreeAssociation/. ↗
  43. Céline Spriet, Etienne Abassi, Jean-Rémy Hochmann, and Liuba Papeo. Visual object categorization in infancy. Proceedings of the National Academy of Sciences, 119(8):e2105866119, 2022. doi: 10.1073/pnas. 2105866119. — [doi:10.1073/pnas](https://doi.org/10.1073/pnas) find
  44. Martin N. Hebart, Adam H. Dickter, Alexis Kidder, Wan Y. Kwok, Anna Corriveau, Caitlin Van Wicklin, and Chris I. Baker. THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PLOS ONE, 14(10):e0223792, 2019. ISSN 1932-6203. doi: 10.1371/journal.pone.0223792. — [doi:10.1371/journal.pone.0223792](https://doi.org/10.1371/journal.pone.0223792) find
  45. Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, and Chris I Baker. THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. eLife, 12:e825… — [doi:10.7554/elife.82580](https://doi.org/10.7554/elife.82580) find
  46. Charles Y. Zheng, Francisco Pereira, Chris I. Baker, and Martin N. Hebart. Revealing interpretable object representations from human behavior. In Proceedings of the 7th International Conference on Learning Representations, 2018. doi: 10.48550/arXiv.1901.02915. — [doi:10.48550/arxiv.1901.02915](https://doi.org/10.48550/arxiv.1901.02915) find
  47. Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. Representational similarity analysis - connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2, 2008. ISSN 1662-5137. doi: 10.3389/neuro.06.004.2008. — [doi:10.3389/neuro.06.004.2008](https://doi.org/10.3389/neuro.06.004.2008) find
  48. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, 2021.
  49. Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision, 2021.
  50. Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A Foundational Language And Vision Alignment Model, 2022. find
  51. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation, 2022.
  52. Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, and Nan Duan. BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning, 2024. find
  53. Romain Beaumont. Large scale openCLIP: L/14, H/14 and g/14 trained on LAION-2B, 2022. URL https://laion.ai/blog/large-openclip. ↗
  54. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023. URL https://arxiv.org/abs/2303.15343. ↗
  55. Noah D. Goodman and Michael C. Frank. Pragmatic Language Interpretation as Probabilistic Inference. Trends in Cognitive Sciences, 20(11):818–829, 2016. ISSN 1364-6613. doi: 10.1016/j.tics.2016.08.005. find
  56. Evan Kidd and Rowena Garcia. How diverse is child language acquisition research? First Language, 42 (6):703–735, 2022. ISSN 0142-7237. doi: 10.1177/01427237211066405. — [doi:10.1177/01427237211066405](https://doi.org/10.1177/01427237211066405) find
  57. Patricia A. Ganea, Megan Bloom Pickard, and Judy S. DeLoache. Transfer between Picture Books and the Real World by Very Young Children. Journal of Cognition and Development, 9(1):46–66, 2008. ISSN 1524-8372. doi: 10.1080/15248370701836592. — [doi:10.1080/15248370701836592](https://doi.org/10.1080/15248370701836592) find
  58. Gabrielle Simcock and Judy DeLoache. Get the picture? The effects of iconicity on toddlers’ reenactment from picture books. Developmental Psychology, 42(6):1352–1357, 2006. ISSN 1939-0599, 0012-1649. doi: 10.1037/0012-1649.42.6.1352. — [doi:10.1037/0012-1649.42.6.1352](https://doi.org/10.1037/0012-1649.42.6.1352) find
  59. Medha Tare, Cynthia Chiong, Patricia Ganea, and Judy DeLoache. Less is more: How manipulative features affect children’s learning from picture books. Journal of Applied Developmental Psychology, 31 (5):395–400, 2010. ISSN 0193-3973. doi: 10.1016/j.appdev.2010.06.005. — [doi:10.1016/j.appdev.2010.06.005](https://doi.org/10.1016/j.appdev.2010.06.005) find
  60. Bria Long, Violet Xiang, Stefan Stojanov, Robert Z. Sparks, Zi Yin, Grace E. Keene, Alvin W. M. Tan, Steven Y. Feng, Chengxu Zhuang, Virginia A. Marchman, Daniel L. K. Yamins, and Michael C. Frank. The BabyView dataset: High-resolution egocentric videos of infants’ and young children’s everyday expe… find
  61. Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. Observational Scaling Laws and the Predictability of Language Model Performance, 2024.
  62. Cheng Wang. Calibration in Deep Learning: A Survey of the State-of-the-Art, 2024. find
  63. Michael C. Frank. Openly accessible LLMs can help us to understand human cognition. Nature Human Behaviour, 7(11):1825–1827, 2023. ISSN 2397-3374. doi: 10.1038/s41562-023-01732-4. — [doi:10.1038/s41562-023-01732-4](https://doi.org/10.1038/s41562-023-01732-4) find
  64. Michael C Frank. Learning variability network exchange (levante): A global framework for measuring children’s learning variability through collaborative data sharing, Sep 2024. URL osf.io/preprints/ psyarxiv/namx2. find
  65. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 740–755, Cham, 2014.…
  66. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scal… — LAION-5B: An open large-scale dataset for training next generation image-text models
  67. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://llava-vl.github.io/ blog/2024-01-30-llava-next/. ↗
  68. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
  69. Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models, 2024. URL https://arxiv.org/abs/2402.14289. — [arXiv:2402.14289](https://arxiv.org/abs/2402.14289) ↗
  70. Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world, 2023. URL https://arxiv.org/abs/2306. 14824. ↗
  71. moondream AI team. moondream2, 2024. URL https://github.com/vikhyat/moondream. ↗
  72. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2024. URL https://arxiv.org/abs/2311.03079. *Table 4: Particip… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tan-2024-devbench-multimodal-developmental-benchmark-language/