Elamur Research⌕
☰
home / papers / 2026 / Minos: A Multimodal Evaluation Model for Bidirectional Generation Between Image and Text

Minos: A Multimodal Evaluation Model for Bidirectional Generation Between Image and Text

Paper · 2026 · Junzhe Zhang, Huixuan Zhang, Xinyu Hu et al.

Summary

Automatic judging of multimodal outputs is still unreliable. Classic metrics often disagree with human ratings, need references, or only work for one task. Existing MLLM-based judges such as LLaVA-Critic and UnifiedReward train on large but loosely filtered data and do not handle both image-to-text (I2T) and text-to-image (T2I) generation equally well.

The authors built Minos-57K from 15 datasets covering 6 task types in both directions. Some data comes from human-rated sources (Polaris, LAVE, ImageReward), and the rest was newly annotated using task-specific guidelines. GPT-4o wrote several candidate analyses and 1–5 scores for each sample. The team then removed inconsistent candidates and rebalanced score distributions. A stricter filter produced a preference set, Minos-DPO-5.8K. They fine-tuned Qwen3-VL-8B on Minos-57K and then aligned it with DPO.

The resulting judge, Minos, gives reference-free scores with explanations. It is the best open-source evaluator on 16 out-of-domain benchmarks and holds up against closed models, despite using under half the training data of earlier work. The results suggest that data quality matters as much as data volume when training evaluation models.

Key points

  • Resources: Minos-57K (SFT data) and Minos-DPO-5.8K (preference pairs), with code at github.com/reroze/MINOS.
  • Data cleaning: Quality control works at two levels. Consistency Filtering acts on individual samples, Score Balancing acts across each dataset, and Delta Score Filtering selects the DPO pairs.
  • Joint I2T and T2I training: Training on both directions helped each one, raising the average Pearson-r from 38.4 (base Qwen3-VL-8B) to 40.9. Training on one direction alone hurt the other; for example, T2I-only data lowered the I2T average from 36.7 to 25.4.
  • Written analyses: Training the model to produce explanations improved agreement with human scores, from 38.8 to 40.9 average Pearson-r.
  • Stated limitations: The work covers only an 8B backbone. The data is a snapshot of currently available human-rated datasets. The authors warn of over-optimization if Minos is used as a reward model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China315 works · 43,268 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Junzhe Zhang, Huixuan Zhang, Xinyu Hu, Li Lin, Mingqi Gao, Shi Qiu and Xiaojun Wan. 2026. Minos: A Multimodal Evaluation Model for Bidirectional Generation Between Image and Text. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.744

@inproceedings{zhang2026minos,
  title = {{Minos: A Multimodal Evaluation Model for Bidirectional Generation Between Image and Text}},
  author = {Junzhe Zhang, Huixuan Zhang, Xinyu Hu, Li Lin, Mingqi Gao, Shi Qiu and Xiaojun Wan},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.744},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.744}
}

References

47 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019. nocaps: novel object captioning at scale. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), page 8947–8956. IEEE. find
  2. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. Preprint, arXiv:2308.12966.
  3. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, and 1 others. 2025. Qwen3-vl technical report. Preprint, arXiv:2511.21631.
  4. Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Pan Zhou, Yao Wan, and Lichao Sun. 2024. Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. arXiv preprint arXiv:2402.04788.
  5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 other…
  6. Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10578–10587. find
  7. Wentao Ge, Shunian Chen, Guiming Chen, Junying Chen, Zhihong Chen, Shuo Yan, Chenghao Zhu, Ziyue Lin, Wenya Xie, Xidong Wang, and 1 others. 2023. Mllm-bench, evaluating multi-modal llms using gpt-4v. arXiv preprint arXiv:2311.13951. — [arXiv:2311.13951](https://arxiv.org/abs/2311.13951) find
  8. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2022. Clipscore: A reference-free evaluation metric for image captioning. Preprint, arXiv:2104.08718.
  9. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore. find
  10. Xinyu Hu, Li Lin, Mingqi Gao, Xunjian Yin, and Xiaojun Wan. 2024. Themis: A reference-free NLG evaluation language model with flexibility and interpretability. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15924–15951, Miami, Florida, USA. Associati… find
  11. Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2022. Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.09699. — [arXiv:2211.09699](https://arxiv.org/abs/2211.09699) find
  12. Jiaxing Huang and Jingyi Zhang. 2024. A survey on evaluation of multimodal large language models. Preprint, arXiv:2408.15769. — [arXiv:2408.15769](https://arxiv.org/abs/2408.15769) find
  13. Dongfu Jiang, Max Ku, Tianle Li, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen. 2024. Genai arena: An open evaluation platform for generative models. arXiv preprint arXiv:2406.04485. — [arXiv:2406.04485](https://arxiv.org/abs/2406.04485) find
  14. Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. 2023a. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Preprint, arXiv:2305.01569. find
  15. Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. 2023b. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in neural information processing systems, 36:36652–36663. find
  16. Lingpeng Kong, and Qi Liu. 2024b. Vlfeedback: A large-scale ai feedback dataset for large vision-language models alignment. arXiv preprint arXiv:2410.09421. find
  17. Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, Junjie Ke, Krishnamurthy Dj Dvijotham, Katie Collins, Yiwen Luo, Yang Li, Kai J Kohlhoff, Deepak Ramachandran, and Vidhya Navalpakkam. 2024. Rich human feedback fo… — [arXiv:2312.10240](https://arxiv.org/abs/2312.10240) find
  18. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer.
  19. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning.
  20. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  21. Oscar Mañas, Benno Krojer, and Aishwarya Agrawal. 2024. Improving automatic vqa evaluation using large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4171–4179. find
  22. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. find
  23. Van-Quang Nguyen, Masanori Suganuma, and Takayuki Okatani. 2022. Grit: Faster and better image captioning transformer using dual visual features. In Computer Vision – ECCV 2022, pages 167–184, Cham. Springer Nature Switzerland. find
  24. Seongyun Lee, Seungone Kim, Sue Park, Geewook Kim, and Minjoon Seo. 2024. Prometheus-vision: Vision-language model as a judge for fine-grained evaluation. In Findings of the Association for Computational Linguistics ACL 2024, pages 11286–11315. find
  25. OpenAI. 2023. GPT-4V(ision) System Card. find
  26. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. Llava-onevision: Easy visual task transfer. Preprint, arXiv:2408.03326.
  27. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.
  28. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR.
  29. Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Bala… — GPT-4 Technical Report
  30. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  31. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728– 53741.
  32. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  33. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048–2057. PMLR.
  34. Yuiga Wada, Kanta Kaneda, Daichi Saito, and Komei Sugiura. 2024. Polos: Multimodal metric learning from human feedback for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13559–13568. find
  35. Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5579–5588. find
  36. Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International conference on machine learning, pages 23318–233…
  37. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, and 1 others. 2024. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37:121475–121499.
  38. Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. 2025. Unified reward model for multimodal understanding and generation. Preprint, arXiv:2503.05236. — [arXiv:2503.05236](https://arxiv.org/abs/2503.05236) find
  39. Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. 2023. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long pap… find
  40. Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. 2023. Gpt-4v(ision) as a generalist evaluator for vision-language tasks. Preprint, arXiv:2311.01361. — [arXiv:2311.01361](https://arxiv.org/abs/2311.01361) find
  41. Yi-Fan Zhang, Xingyu Lu, Xiao Hu, Chaoyou Fu, Bin Wen, Tianke Zhang, Changyi Liu, Kaiyu Jiang, Kaibing Chen, Kaiyu Tang, Haojie Ding, Jiankang Chen, Fan Yang, Zhang Zhang, Tingting Gao, and Liang Wang. 2025. R1-reward: Training multimodal reward model through stable reinforcement learning. Preprint,… — [arXiv:2505.02835](https://arxiv.org/abs/2505.02835) find
  42. A We present the full name of each task mentioned in Table 3. The full name remains the same as (Chen et al., 2024). find
  43. Task Name (Short) Task Name (Full) CO C.C. Dif Graph Math Text WIT Chart Vis CC M2W Sci Aes MM MS COCO Conceptual Captions DiffusionDB InfographicVOA MathVista TextVOA WIT ChartOA VisIT-Bench CC-3M Concept-balanced Mind2Web ScienceOA AesBench MMvet Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen,… — [arXiv:2306.09341](https://arxiv.org/abs/2306.09341) find
  44. Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang, Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li, Lijuan Wang, and Huaxiu Yao. 2024. Mmie: Massive multimodal interleaved comprehension benchmark for large vision-language models. arXiv preprint arXiv:2410.10139. — [arXiv:2410.10139](https://arxiv.org/abs/2410.10139) find
  45. Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. 2024. Llava-critic: Learning to evaluate multimodal models. find
  46. Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. 2023. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903– 15935. find
  47. Experimental Details *Table 8: The full name of each task.* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-minos-multimodal-evaluation-model-bidirectional/