DREAM: A Multicultural Multimodal Dataset Linking Dialogues and Realistic Image Sequences
Summary
Multimodal dialogue corpora rarely combine conversations with image sequences that keep each speaker's appearance stable across turns, and they seldom control demographic diversity in a systematic way. DREAM addresses this gap. It is a synthetic resource of 1,000 persona-grounded dialogues taken from PersonaChat and ComperDial. Each speaker gets an expanded profile with structured and narrative layers, plus a photorealistic portrait. Selected dialogue turns are then illustrated as storyboard-style scenes, using the portraits as identity anchors. The authors built a model-agnostic, batchable pipeline: o4-mini handles persona expansion and gpt-image-1 renders portraits and scenes. Prompts are culturally aware, and turn selection targets visually salient moments. Everything is packaged in one JSON schema. Quality is assessed in two ways: human ratings of realism, coherence, consistency and perceived demographics, and automated portrait analysis with GPT-5. Annotators' comments are grouped into eight recurring failure categories, such as identity drift and anatomy errors. The resource enables studies of visual grounding, character continuity, and bias in appearance-based perception, while avoiding real personal data.
Key points
- The corpus has 1,000 dialogues: 900 from PersonaChat (train_both_revised.txt, out of 8,939 available) and 100 from ComperDial (its full public set). Each dialogue has two portraits and a set of scene images.
- Each persona combines two layers:
- profile_struct: age, gender_identity, sexual_orientation, a 10-cluster appearance-based ethnicity, nationality, and residence_country. Values come from textual evidence where available, otherwise from controlled fallback distributions.
- profile_narrative: five sections of descriptive text.
- Nationality and residence are kept separate, which allows diaspora-like profiles. Visual turns are tagged with a scene type (shared, memory, imagined, cutaway, montage) and a speaker focus.
- The generation stack uses o4-mini for text and gpt-image-1 for images. GPT-5 serves as the vision–language evaluator and returns strict JSON labels for ethnicity, gender and age.
- Free-text human feedback falls into eight error types: anatomy, object physics, text/iconography, identity drift, wardrobe continuity, gaze/interaction, scene–dialogue mismatch, and object semantics.
Cite
@inproceedings{mallo2026dream,
title = {{DREAM: A Multicultural Multimodal Dataset Linking Dialogues and Realistic Image Sequences}},
author = {Mallo, Juan and Estecha-Garitagoitia, Marcos and Cordoba, Ricardo and D'Haro, Luis Fernando},
booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
year = {2026},
doi = {10.63317/2v7b4xhs2d5g},
url = {https://doi.org/10.63317/2v7b4xhs2d5g}
}References
- American Psychological Association. 2023. Gender identity (apa dictionary of psychology). https://dictionary.apa.org/gender-ident ity. ↗
- Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR. find our card
- Emily M. Bender and Batya Friedman. 2018. Data statements for NLP: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
- Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023. Promptify: Text-to-image generation through interactive prompt exploration with large language models. In UIST. find
- Centers for Disease Control and Prevention. 2024. Preferred terms for select population groups & communities. https://www.cdc.gov/heal th-communication/php/toolkit/prefe rred-terms.html. ↗
- Abhishek Das, Satwik Kottur, Khushi Gupta, Amanpreet Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In CVPR. find
- Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In CVPR. find
- Gallup. 2023. Lgbt identification in the u.s. https://news.gallup.com/poll/656708/lgbt q-identification-rises.aspx. ↗
- Mao Xun Huang and Hen-Hsen Huang. 2024. Integrating text-to-image and vision language models for synergistic dataset generation. In IJCAI. find
- Office for National Statistics. 2023. Sexual orientation, uk: annual estimates. https://www.on s.gov.uk/peoplepopulationandcommun ity/culturalidentity/sexuality/bul letins/sexualidentityuk/2023. ↗
- Kurt Shuster, Samuel Humeau, Antoine Bordes, and Jason Weston. 2018. Image chat: Engaging grounded conversations. In ACL. find
- United Nations Department of Economic and Social Affairs. 2020. International migrant stock. https://www.un.org/development/desa/pd/. ↗
- United Nations Statistics Division. 2024. Standard country or area codes for statistical use (m49). https://unstats.un.org/unsd/method ology/m49/. ↗
- Hiromi Wakaki, Yuki Mitsufuji, Yoshinori Maeda, Yukiko Nishimura, Silin Gao, Mengjie Zhao, Keiichi Yamada, and Antoine Bosselut. 2024. Comperdial: Commonsense persona-grounded dialogue dataset and benchmark. arXiv preprint arXiv:2406.11228. — [arXiv:2406.11228](https://arxiv.org/abs/2406.11228) find
- Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243. 12. Language Resource References Awsaf49. 2018. PersonaChat (Hugging Face mirror) train_both_revised.txt.… find our card
This page in Markdown → https://md.research.elamur.ai/free/papers/mallo-2026-dream-multicultural-multimodal-dataset-linking/