Elamur Research⌕
☰
home / papers / 2026 / DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents

DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents

Paper · 2026 · Kai Shi, Jun Yang, Ni Yang et al.

Summary

Phone assistants built on multimodal LLMs must combine several skills, including planning, tool calling, memory and screen understanding. Multitask SFT is the usual way to teach these skills, but choosing how much of each training dataset to include is expensive to get right. Existing mixture optimizers target pretraining loss, and simple sampling or grid search scale poorly. The authors from OPPO AI Center propose DaMo. It samples a modest number of mixture ratios, fine-tunes a small MLLM on each, and records downstream benchmark scores. An MLP is then trained to map ratios to scores, and the best mixture is found by extrapolating across the full mixing space. They also release PhoneAgentBench, an industrial-style benchmark for mobile agent tasks that goes beyond GUI interaction. DaMo predicts scores well and produces better-performing models than uniform, natural, random and DML mixtures at a similar compute budget. Its chosen mixtures also transfer to other model families. This makes data-ratio tuning for fine-tuning MLLMs more principled and cheaper to reuse.

Key points

  • PhoneAgentBench: 1,235 validated QA pairs organized into six datasets, such as MT-Plan and Mobile-FC.
    • It covers four capabilities: task planning, device-native tool use, multimodal memory and screen context understanding.
    • It draws on 50 core mobile interfaces.
    • MT-Plan's complexity score is about 5.4× that of T-Eval.
  • Prediction accuracy: DaMo reaches R²=0.81 in small-scale pilot runs, with InternVL2.5-4B as the base model.
  • Gains over DML:
    • +3.06% average across PhoneAgentBench, BFCL-v3, MME-Reasoning, MME-Perception and OCRBench.
    • +6.70% when optimizing only for open-source benchmarks.
    • BFCL-v3 rises from 34.69% (Uniform) to 47.43% when DaMo optimizes for it alone.
  • Cost: DaMo uses about 4,242 H20 GPU-hours, roughly the same as DML. Uniform and Natural mixtures did not improve even when given longer training.
  • Transfer to other models: The tested targets were Qwen2.5VL-3B/7B and InternVL3-14B.
    • Reusing the chosen mixture directly yields +0.31–2.87%.
    • Adding a linear correction fitted on 20 calibration samples yields +1.61–3.51%.
    • Correlation between predicted and actual scores rises from r > 0.75 to r > 0.9 with the correction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Kai Shi, Jun Yang, Ni Yang, Binqiang Pan, Qingsong Xie, Zhangchao, Zhenyu Yang, Tianhuang Su and Haonan Lu. 2026. DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1439

@inproceedings{shi2026damo,
  title = {{DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents}},
  author = {Kai Shi, Jun Yang, Ni Yang, Binqiang Pan, Qingsong Xie, Zhangchao, Zhenyu Yang, Tianhuang Su and Haonan Lu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1439},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1439}
}

References

66 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abhishek Shrivastava. 2023. sentiment-analysis-dataset. https://www.kaggle.com/datasets/ abhi8923shriv/sentiment-analysis-dataset. Accessed: 2025-03-12. ↗
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  3. Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. 2023. Efficient online data mixing for language model pre-training. arXiv preprint arXiv:2312.02406.
  4. Martin Anthony and Peter L Bartlett. 1999. Neural network learning: Theoretical foundations. Cambridge University Press. find
  5. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. Qwen3-vl technical report. arXiv prepr…
  6. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025b. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  7. Kinjal Basu. 2024. Bridging knowledge gaps in llms via function calls. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 5556–5557. find
  8. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935.
  9. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and 1 others. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394.
  10. Longxi Gao, Li Zhang, Shihe Wang, Shangguang Wang, Yuanchun Li, and Mengwei Xu. 2024. Mobileviews: A large-scale mobile gui dataset. arXiv preprint arXiv:2409.14337. — [arXiv:2409.14337](https://arxiv.org/abs/2409.14337) find
  11. Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, and Bolin Ding. 2024. Bimix: Bivariate data mixing law for language model pretraining. arXiv preprint arXiv:2405.14908. — [arXiv:2405.14908](https://arxiv.org/abs/2405.14908) find
  12. Longling Geng and Edward Y Chang. 2025. Realm-bench: A real-world planning benchmark for llms and multi-agent systems. arXiv preprint arXiv:2502.18836. — [arXiv:2502.18836](https://arxiv.org/abs/2502.18836) find
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  14. Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, Zhenchong Hu, Bo-Wen Zhang, Jijie Li, Dong Liang, Yingli Zhao, Yulong Ao, Yaoqi Liu, Fangxiang Feng, and Guang Liu. 2024. Infinity-mm: Scaling multimodal performance with la…
  15. David Castillo-Bolado, Joseph Davidson, Finlay Gray, and Marek Rosa. 2024. Beyond prompts: Dynamic conversational benchmarking of large language models. Preprint, arXiv:2409.20222. — [arXiv:2409.20222](https://arxiv.org/abs/2409.20222) find
  16. Hamed Rahimi. 2024. Sujetfinancevision10k. https://huggingface.co/datasets/sujet-ai/ Sujet-Finance-Vision-10k. Accessed: 2025-03-12. ↗
  17. Mingyang Chen, Haoze Sun, Tianpeng Li, Fan Yang, Hao Liang, Keer Lu, Bin Cui, Wentao Zhang, Zenan Zhou, and Weipeng Chen. 2024a. Facilitating multi-turn function calling for llms via compositional instruction tuning. arXiv preprint arXiv:2410.12952. — [arXiv:2410.12952](https://arxiv.org/abs/2410.12952) find
  18. Chen Huang, Shuangfei Zhai, Walter Talbott, Miguel Angel Bautista, Shih-Yu Sun, Carlos Guestrin, and Josh Susskind. 2019. Addressing the loss-metric mismatch with adaptive loss alignment. Preprint, arXiv:1905.05895. — [arXiv:1905.05895](https://arxiv.org/abs/1905.05895) find
  19. Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, and 1 others. 2023. T-eval: Evaluating the tool utilization capability of large language models step by step. arXiv preprint arXiv:2312.14033. find
  20. Tim Ingold. 2021. The perception of the environment: essays on livelihood, dwelling and skill. routledge. find
  21. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, and 1 others. 2024b. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271.
  22. Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. 2024. Scaling laws for downstream task performance of large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models. find
  23. Andrei N Kolmogorov and Vladimir M Tikhomirov. 1959. ε-entropy and ε-capacity of sets in function spaces. Uspekhi Matematicheskikh Nauk, 14(2):3– 86. find
  24. Koushyar Komeilizadeh, Arne Kaps, and Fabian Dud-deck. 2023. Isovolumetric adaptations to space-filling design of experiments. Optimization and Engineering, 24(2):1267–1288. find
  25. Hao Li, Chenghao Yang, An Zhang, Yang Deng, Xiang Wang, and Tat-Seng Chua. 2024a. Hello again! llm-powered personalized agent for long-term dialogue. arXiv preprint arXiv:2406.05925.
  26. Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. 2025a. Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.07981. — [arXiv:2504.07981](https://arxiv.org/abs/2504.07981) find
  27. Lingyu Li, Yixu Wang, Haiquan Zhao, Shuqi Kong, Yan Teng, Chunbo Li, and Yingchun Wang. 2024b. Reflection-bench: probing ai intelligence with reflection. arXiv preprint arXiv:2410.16270. — [arXiv:2410.16270](https://arxiv.org/abs/2410.16270) find
  28. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. Preprint, arXiv:2304.08244.
  29. Yuan Li, Zhengzhong Liu, and Eric Xing. 2025b. Data mixing optimization for supervised fine-tuning of large language models. arXiv preprint arXiv:2508.11953. — [arXiv:2508.11953](https://arxiv.org/abs/2508.11953) find
  30. Liang Xu. 2024. Superclue-agent. https://github.com/CLUEbenchmark/SuperCLUE-Agent. Accessed: 2025-03-12. ↗
  31. P Liashchynskyi. 2019. Grid search, random search, genetic algorithm: A big comparison for nas. arXiv preprint ArXiv:1912.06059. — [arXiv:1912.06059](https://arxiv.org/abs/1912.06059) find
  32. Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. 2024a. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492.
  33. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, and 1 others. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688.
  34. Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, and 1 others. 2024b. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06327. — [arXiv:2408.06327](https://arxiv.org/abs/2408.06327) find
  35. Yanming Liu, Xinyue Peng, Jiannan Cao, Shi Bo, Yuwei Zhang, Xuhong Zhang, Sheng Cheng, Xun Wang, Jianwei Yin, and Tianyu Du. 2024c. Tool-planner: Task planning with clusters across multiple tools. arXiv preprint arXiv:2406.03807. — [arXiv:2406.03807](https://arxiv.org/abs/2406.03807) find
  36. Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. 2024d. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12).
  37. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521. find
  38. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.
  39. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. Preprint, arXiv:2311.12983.
  40. Paul Michel, Sebastian Ruder, and Dani Yogatama. 2021. Balancing average and worst-case accuracy in multitask learning. arXiv: Learning,arXiv: Learning. minyang. 2024. invoices-and-receipts_ocr_v1. https://huggingface.co/datasets/mychen76/ invoices-and-receipts_ocr_v1. Accessed: 2025-03-12. ↗
  41. Niccolò Zanichelli. 2024. arxiv-ocr-v0.1.2. https://huggingface.co/datasets/nz/ arxiv-ocr-v0.1.2. Accessed: 2025-03-12. ↗
  42. Haojie Pan, Zepeng Zhai, Hao Yuan, Yaojia Lv, Ruiji Fu, Ming Liu, Zhongyuan Wang, and Bing Qin. 2023. Kwaiagents: Generalized information-seeking agent system with large language models. arXiv preprint arXiv:2312.04889. — [arXiv:2312.04889](https://arxiv.org/abs/2312.04889) find
  43. Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025a. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning. find
  44. Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025b. The berkeley function calling leaderboard (bfcl) v4: From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning. find
  45. Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. 2025c. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning. qgyd2… find
  46. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. Toolllm: Facilitating large language models to master 16000+ real…
  47. Shiori Sagawa, PangWei Koh, Tatsunori Hashimoto, and Percy Liang. 2020. Distributionally robust neural networks. International Conference on Learning Representations,International Conference on Learning Representations. shibing624. 2023. sharegpt_gpt4. https://huggingface.co/datasets/shibing624/ sh… ↗
  48. Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2998–3009.
  49. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems, 36:38975–38987.
  50. Martin J Wainwright. 2019. High-dimensional statistics: A non-asymptotic viewpoint. Cambridge University Press. See Chapter 5 on Metric Entropy and Covering Numbers. find
  51. Xuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang, Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, and 1 others. 2025. Mmbench-gui: Hierarchical multi-platform evaluation framework for gui agents. arXiv preprint arXiv:2507.19478. — [arXiv:2507.19478](https://arxiv.org/abs/2507.19478) find
  52. Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. 2024. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning. arXiv preprint… — [arXiv:2401.06805](https://arxiv.org/abs/2401.06805) find
  53. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2023a. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798–69818. find
  54. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2023b. Doremi: Optimizing data mixtures speeds up language model pretraining. In Advances in Neural Information Processing Systems, volume 36, pages 69798–69818. Curran As… find
  55. Shiming Xie, Hong Chen, Fred Yu, Zeye Sun, and Xiuyu Wu. 2024. Minor sft loss for llm fine-tune to increase performance and reduce model deviation. arXiv preprint arXiv:2408.10642. — [arXiv:2408.10642](https://arxiv.org/abs/2408.10642) find
  56. Dmitry Yarotsky. 2017. Error bounds for approximations with deep relu networks. Neural Networks, 94:103–114. find
  57. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. 2024. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952. find
  58. Jiakang Yuan, Tianshuo Peng, Yilei Jiang, Yiting Lu, Renrui Zhang, Kaituo Feng, Chaoyou Fu, Tao Chen, Lei Bai, Bo Zhang, and 1 others. 2025. Mme-reasoning: A comprehensive benchmark for logical reasoning in mllms. arXiv preprint arXiv:2505.21327. — [arXiv:2505.21327](https://arxiv.org/abs/2505.21327) find
  59. Ruifeng Yuan, Shichao Sun, Yongqi Li, Zili Wang, Ziqiang Cao, and Wenjie Li. 2023. Personalized large language model assistant with evolving conditional memory. arXiv preprint arXiv:2312.17257. find
  60. Yuanhan Zhang Bo Li Songyang Zhnag Wangbo Zhao Yike Yuan Jiaqi Wang Conghui He Ziwei Liu Kai Chen Dahua Lin Yuan Liu, Haodong Duan. 2023. Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06281.
  61. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. Mmmu: A massive multi-discipline multimodal u…
  62. Sewall Wright. 1921. Correlation and causation. Journal of agricultural research, 20(7):557. find
  63. Jin Zhang, Jianyang Xue, and Bochao Cao. 2024. Improving agent performance in fluid environments by perceptual pretraining. Physics of Fluids, 36(12). find
  64. Yuxin Xiao, Shujian Zhang, Wenxuan Zhou, Marzyeh Ghassemi, and Sanqiang Zhao. 2024. Sftmix: Elevating language model instruction tuning with mixup recipe. arXiv preprint arXiv:2410.05248.
  65. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, and 1 others. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479.
  66. Tong Zhu, Daize Dong, Xiaoye Qu, Jiacheng Ruan, Wenliang Chen, and Yu Cheng. 2024. Dynamic data mixing maximizes instruction tuning for mixture-of-experts. arXiv preprint arXiv:2406.11256. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2026-damo-data-mixing-optimizer-fine/