Elamur Research⌕
☰
home / papers / 2026 / MUSIA: Multilingual Story Illustration Corpus for Cross-Cultural Alignment and Generation

MUSIA: Multilingual Story Illustration Corpus for Cross-Cultural Alignment and Generation

Summary

Most story-illustration research uses English-only datasets built around Western imagery. As a result, text-to-image systems often give non-Western narratives foreign clothing, architecture and symbols. The authors address this with MUSIA, a bilingual English–Hindi resource of stories from open literary and folk sources, including folktales, moral tales and children's literature. Each story is linked to several illustrations matched scene by scene, and the text–image mappings are checked for quality. To give the corpus a reproducible reference point, they build a simple two-stage baseline: a transformer model condenses the narrative into semantic summaries, which then serve as prompts for a diffusion image generator. Human judges scored the outputs on relevance, visual quality and consistency, and the baseline performed well on all three. Qualitative examples show stable characters across frames and backgrounds suited to the regional setting. The work matters because it is an early, openly licensed benchmark for culturally grounded visual storytelling in Indian and other underrepresented languages, and it is planned to grow through versioned releases.

Key points

  • Coverage and licence: The current release (MUSIA-2025) covers English and Hindi narrative stories from open-access or public-domain material. Content was screened to remove violent or religiously sensitive items, and the dataset is released under CC BY-NC 4.0 for research and non-commercial use only. The supplied text does not state the corpus size.
  • Baseline: Transformer-based semantic summarization feeds a diffusion model. Prompts include a "cartoon style" descriptor to keep the visual style uniform.
  • Human evaluation: Four annotators rated each full illustration sequence on a 3-point Likert scale (1 = Fair, 2 = Moderate, 3 = Good), and their scores were averaged. The three dimensions were:
    • Relevance (text→image),
    • Visual Quality,
    • Consistency (image→image).
  • Origin: MUSIA extends the corpus of the Multilingual Story Illustration Shared Task at FIRE, which the authors organized. Results by approach are reported in Tables 4 (English) and 5 (Hindi). In that task, pipelines that used LLMs to build prompts for diffusion models did best, but keeping character identity consistent remained difficult.
  • Plans: The authors intend to add Bengali, Marathi and Dravidian languages, as well as poems, comics and folk narratives. They also plan to test instruction-tuned MLLMs with RLHF, and every subsequent release will be locked in place and accompanied by documentation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3pk9sab3nfuo

@inproceedings{tewari2026musia,
  title = {{MUSIA: Multilingual Story Illustration Corpus for Cross-Cultural Alignment and Generation}},
  author = {Tewari, Krishna and Chanda, Supriya and Patil, Nirmit and Pal, Sukomal},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3pk9sab3nfuo},
  url = {https://doi.org/10.63317/3pk9sab3nfuo}
}

References

33 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  2. Hong Chen, Rujun Han, Te-Lin Wu, Hideki Nakayama, and Nanyun Peng. 2022. Character-centric story visualization via visual planning and token alignment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8259–8272, Abu Dhabi, United Arab Emirates. Associa… find
  3. Prafulla Dhariwal and Alex Nichol. 2021. Diffusion models beat gans on image synthesis. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY, USA. Curran Associates Inc. find
  4. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In Intern… find
  5. Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. 2022. Make-a-scene: Scene-based text-to-image generation with human priors. In Proceedings of the European Conference on Computer Vision (ECCV), pages 604–620. Springer. find
  6. Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. 2022. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10696– 10706. find
  7. Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. 2018. Imagine this! scripts to compositions to videos. In Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII, page 610–626, Berlin, Heidelberg. Springer-V… find
  8. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2023. Do androids laugh at electric sheep? humor "understanding" benchmarks from the new yorker caption contest. Transactions of the Association for Computational Linguistics, 11:688–705. find
  9. Xudong Hong, Vera Demberg, Asad Sayeed, Qiankun Zheng, and Bernt Schiele. 2023. Visual coherence loss for coherent and visually grounded story generation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9456–9470, Toronto, Canada. Association for Computational Linguisti… find
  10. Junjie Hu, Yu Cheng, Zhe Gan, Jingjing Liu, Jianfeng Gao, and Graham Neubig. 2024. Visual storytelling dataset (vist). https://service.tib.eu/ldmservice/dataset/visual-storytelling-dataset--vist-. Accessed: 2025-10-25. ↗
  11. Ankit Jain, Anand Mishra, and C. V. Jawahar. 2021. Understanding cartoon emotion using integrated deep neural network on large dataset. Pattern Recognition Letters, 145:148–154. find
  12. Kishan Kachhadiya and Parth Patel. 2025. Leveraging large language model (llm) and v-llm for zero-shot multilingual story illustration. In Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2025), volume 4173 of CEUR Workshop Proceedings, pages 113–120. CEUR-WS.org. find
  13. Janak Kapuriya and Paul Buitelaar. 2025. Flintstonessv++: Improving story narration using visual scene graph. In Proceedings of the 8th Workshop on Narrative Extraction From Texts (Text2Story 2025), volume 3964 of CEUR Workshop Proceedings. Accessed: 2025-10-25. find
  14. Chunye Li, Liya Kong, and Zhiping Zhou. 2020. Improved-storygan for sequential images visualization. Journal of Visual Communication and Image Representation, 73:102956. find
  15. Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao. 2019. StoryGAN: A Sequential Conditional GAN for Story Visualization . In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6322–6331, Los Alamitos, CA… find
  16. Adyasha Maharana, Darryl Hannan, and Mohit Bansal. 2022. Storydall-e: Adapting pretrained text-to-image transformers for story continuation. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVII, page 70–87, Berlin, Heidelberg. Spri… find
  17. Shazia Mannan, Asha Hegde, and Sharal Coelho. 2025. Bridging cultures through ai: The art of multilingual storytelling. In Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2025), volume 4173 of CEUR Workshop Proceedings, pages 139–143. CEUR-WS.org. find
  18. Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2022. Glide: Photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 1… find
  19. Daniel A. P. Oliveira, Eugénio Ribeiro, and David Martins de Matos. 2025. Story generation from visual inputs: Techniques, related tasks, and challenges. Information, 16(9). find
  20. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th Internatio… find our card
  21. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents. find our card
  22. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821–883… find our card
  23. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models . In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674–10685, Los Alamitos, CA, USA. IEEE Computer Society. find
  24. Mrinmoy Sadhukhan, Indrajit Bhattacharya, and Paramartha Dutta. 2025. Narrart: Multilingual story illustration with ai for english and hindi narratives. In Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2025), volume 4173 of CEUR Workshop Proceedings, pages 130–138. CEUR-WS.or… find
  25. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Lit, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Raphael Gontijo-Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. 2022. Photorealistic text-to-image diffusion models with deep… find our card
  26. Sharma Nandini Surendra and Divya. 2025. Multilingual story illustration for musia 2025 using one-prompt-one-story image generation. In Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2025), volume 4173 of CEUR Workshop Proceedings, pages 121–129. CEUR-WS.org. find
  27. Krishna Tewari, Anshita Malviya, Supriya Chanda, Arjun Mukherjee, and Sukomal Pal. 2026a. Findings of the shared task on multilingual story illustration: Bridging cultures through ai artistry (musia). In Proceedings of the Forum for Information Retrieval Evaluation (FIRE 2025) Working Notes, volume… find
  28. Krishna Tewari, Anshita Malviya, Supriya Chanda, Arjun Mukherjee, and Sukomal Pal. 2026b. Overview of the shared task on multilingual story illustration: Bridging cultures through ai artistry (musia). In Proceedings of the 17th Annual Meeting of the Forum for Information Retrieval Evaluation, FIRE ’… find
  29. Zilyu Ye, Jinxiu Liu, Ruotian Peng, Jinjin Cao, Zhiyang Chen, Yiyang Zhang, Ziwei Xuan, Mingyuan Zhou, Xiaoqian Shen, Mohamed Elhoseiny, Qi Liu, and Guo-Jun Qi. 2024. Openstory++: A large-scale dataset and benchmark for instance-aware open-domain visual storytelling. arXiv preprint arXiv:2408.03695. — [arXiv:2408.03695](https://arxiv.org/abs/2408.03695) find
  30. Jiahui Yu, Ting Wang, Zhe Zhang, Zhiwei Zhang, et al. 2022. Scaling autoregressive models for content-rich text-to-image generation. In Advances in Neural Information Processing Systems (NeurIPS). find
  31. Yangshu Yuan, Heng Chen, and Christian Ng. 2025. Instruction tuning for story understanding and generation with weak supervision. arXiv preprint arXiv:2501.15574. — [arXiv:2501.15574](https://arxiv.org/abs/2501.15574) find
  32. Gangyan Zeng, Zhaohui Li, and Yuan Zhang. 2019. Pororogan: An improved story visualization model on pororo-sv dataset. Proceedings of the 3rd International Conference on Computer Science and Artificial Intelligence, pages 1–5. find
  33. Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. 2019. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5802–5810. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tewari-2026-musia-multilingual-story-illustration-corpus/