Elamur Research⌕
☰
home / papers / 2026 / ROSCO-Omni: Multimodal LLM-Based Communication Understanding for Non- and Minimally-Speaking Autistic Individuals

ROSCO-Omni: Multimodal LLM-Based Communication Understanding for Non- and Minimally-Speaking Autistic Individuals

Paper · 2026 · Siddhant Bikram Shah and Kristina T. Johnson

Summary

About three in ten autistic people stay non- or minimally-speaking all their lives. They still communicate through gestures, vocalizations, facial expressions and AAC devices, but outsiders often misread these signals. Most machine-learning work on autism targets diagnosis. This paper instead asks what such individuals are trying to say. It splits the task into two labels: communicative actions (the physical channel used) and communicative functions (the pragmatic intent behind it).

The authors analyze the ROSCO dataset. It holds short home videos recorded over Zoom, and the children's own caregivers annotated them with multiple labels per clip. The paper benchmarks open and closed MLLMs on these videos and finds that zero-shot performance is weak and few-shot prompting adds little. To address this, the authors propose ROSCO-Omni, a distillation pipeline: Gemini-3-Flash, guided by the gold labels, writes reasoning-based instruction data, which is then used to fine-tune the open Qwen3-Omni.

The resulting model beats Gemini-2.5-Flash and comes close to its teacher. This suggests open models can be adapted to serve an underserved population, although absolute scores remain modest.

Key points

  • ROSCO dataset:
    • 2,903 clips, averaging about 3.11 s, drawn from 34 sessions.
    • 27 children aged 2–12, all diagnosed with PMS, PHTS or TSC.
    • Annotations cover 6 action and 6 function classes, with up to three actions and two functions per clip.
  • Annotation protocol: Primary caregivers rewatched their sessions with a researcher in 10-second segments. Labels therefore reflect intent as understood by the people who know the child best, rather than a clinician's reading.
  • ROSCO-Omni: A teacher-student setup in which Gemini-3-Flash produces label-guided synthetic instructions and Qwen3-Omni is fine-tuned on them.
  • Evaluation: The benchmark includes per-participant results and challenge sets covering zero-shot generalization, cross-session transfer and sensitivity to instruction type. Results vary strongly across individuals, and class imbalance is an issue (e.g., Gestures appear far more often than Face).
  • Availability: Code is public at github.com/SiddhantBikram/ROSCO-Omni. The data will be released under an IRB-approved protocol.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northeastern UniversityBoston · United States57 works · 6,533 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksNeuroscience · 165 worksCognitive Neuroscience · 164 worksHealth Professions · 69 worksOccupational Therapy · 9 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Siddhant Bikram Shah and Kristina T. Johnson. 2026. ROSCO-Omni: Multimodal LLM-Based Communication Understanding for Non- and Minimally-Speaking Autistic Individuals. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2011

@inproceedings{shah2026rosco,
  title = {{ROSCO-Omni: Multimodal LLM-Based Communication Understanding for Non- and Minimally-Speaking Autistic Individuals}},
  author = {Siddhant Bikram Shah and Kristina T. Johnson},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2011},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2011}
}

References

33 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Masroor Ahmed, Sadam Hussain, Farman Ali, Anna Karen Gárate-Escamilla, Ivan Amaya, Gilberto Ochoa-Ruiz, and José Carlos Ortiz-Bayliss. 2025. Summarizing recent developments on autism spectrum disorder detection and classification through machine learning and deep learning techniques. Applied Science… find
  2. Yuxuan Cai, Jiangning Zhang, Haoyang He, Xinwei He, Ao Tong, Zhenye Gan, Chengjie Wang, Zhucun Xue, Yong Liu, and Xiang Bai. 2025. Llava-kd: A framework of distilling multimodal large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 239–249. find
  3. Yang Chen, Jingcai Guo, Miaoge Li, Zhijie Rao, and Song Guo. 2026. Star++: Region-aware conditional semantics via interpretable side information for zero-shot skeleton action recognition. IEEE Transactions on Circuits and Systems for Video Technology. find
  4. Hyundong Justin Cho, Spencer Lin, Tejas Srinivasan, Michael Saxon, Deuksin Kwon, Natali T Chavez, and Jonathan May. 2025. Can vision language models understand mimed actions? In Findings of the Association for Computational Linguistics: ACL 2025, pages 26744–26759. find
  5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  6. Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.
  7. Shijian Deng, Erin E Kosloski, Siddhi Patel, Zeke A Barnett, Yiyang Nan, Alexander Kaplan, Sisira Aaruka-palli, William T Doan, Matthew Wang, Harsh Singh, and 1 others. 2024. Hear me, see me, understand me: Audio-visual autism behavior recognition. IEEE Transactions on Multimedia. find
  8. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  9. Jiaxing Huang, Jingyi Zhang, Kai Jiang, Han Qiu, Xiaoqin Zhang, Ling Shao, Shijian Lu, and Dacheng Tao. 2025a. Visual instruction tuning towards general-purpose multimodal large language model: A survey. International Journal of Computer Vision, 133(11):8151–8189. find
  10. Shulin Huang, Linyi Yang, Yan Song, Shuang Chen, Leyang Cui, Ziyu Wan, Qingcheng Zeng, Ying Wen, Kun Shao, Weinan Zhang, and 1 others. 2025b. Thinkbench: Dynamic out-of-distribution evaluation for robust llm reasoning. arXiv preprint arXiv:2502.16268. — [arXiv:2502.16268](https://arxiv.org/abs/2502.16268) find
  11. Kristina T Johnson, Jaya Narain, Thomas Quatieri, Pattie Maes, and Rosalind W Picard. 2023. Recanvo: A database of real-world communicative and affective nonverbal vocalizations. Scientific Data, 10(1):523. find
  12. Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. When can llms actually correct their own mistakes? a critical survey of self-correction of llms. Transactions of the Association for Computational Linguistics, 12:1417–1440.
  13. Stephen Wen Hwooi Khor, Aznul Qalid Md Sabri, and Alice Othmani. 2024. Autism classification and monitoring from predicted categorical and dimensional emotions of video features. Signal, Image and Video Processing, 18(1):191–198. find
  14. Harsh Kumar, Ruiwei Xiao, Benjamin Lawson, Ilya Musabirov, Jiakai Shi, Xinyuan Wang, Huayin Luo, Joseph Jay Williams, Anna N Rafferty, John Stamper, and 1 others. 2024. Supporting self-reflection at scale with large language models: Insights from randomized field experiments in classrooms. In Procee… find
  15. Jicheng Li, Vuthea Chheang, Pinar Kullu, Eli Brignac, Zhang Guo, Anjana Bhat, Kenneth E Barner, and Roghayeh Leila Barmaki. 2023. Mmasd: A multimodal dataset for autism intervention analysis. In Proceedings of the 25th International Conference on Multimodal Interaction, pages 397–405. find
  16. Haoxuan Qu, Yujun Cai, and Jun Liu. 2024. Llms are good action recognizers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18395–18406. find
  17. V Rose, David Trembath, Deb Keen, and Jessica Paynter. 2016. The proportion of minimally verbal children with autism spectrum disorder in a community-based early intervention programme. Journal of Intellectual Disability Research, 60(5):464–477. find
  18. Manuel Serna-Aguilera, Xuan Bac Nguyen, Han-Seok Seo, and Khoa Luu. 2024. A novel dataset for video-based autism classification leveraging extra-stimulatory behavior. arXiv preprint arXiv:2409.04598. find
  19. Siddhant Bikram Shah and Kristina T Johnson. 2025a. Multi-feature audio fusion for nonverbal vocalization classification. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  20. Siddhant Bikram Shah and Kristina T Johnson. 2025b. N-core: N-view consistency regularization for disentangled representation learning in nonverbal vocalizations. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33362–33379. find
  21. Siddhant Bikram Shah and Kristina T Johnson. 2026. Axon: Action characterization through cross-modal knowledge distillation for neurodiverse individuals. find
  22. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 39228–39238. find
  23. Helen Tager-Flusberg and Connie Kasari. 2013. Minimally verbal school-aged children with autism spectrum disorder: The neglected end of the spectrum. Autism research, 6(6):468–478. find
  24. Chelsea La Valle, Karen Chenausky, and Helen Tager-Flusberg. 2021. How do minimally verbal children and adolescents with autism spectrum disorder use communicative gestures to complement their spoken language abilities? Autism & Developmental Language Impairments, 6:23969415211035065. find
  25. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  26. Wangmeng Xiang, Chao Li, Yuxuan Zhou, Biao Wang, and Lei Zhang. 2023. Generative action description prompts for skeleton-based action recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10276–10285. find
  27. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025. Qwen3-omni technical report. Preprint, arXiv:2509.17765.
  28. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  29. Shijia Yang, Bohan Zhai, Quanzeng You, Jianbo Yuan, Hongxia Yang, and Chenfeng Xu. 2024. Law of vision representation in mllms. arXiv preprint arXiv:2408.16357. — [arXiv:2408.16357](https://arxiv.org/abs/2408.16357) find
  30. Ziqian Yang, Yuyao Zhang, Jiachuan Ning, Xin Wang, and Zhihui Wu. 2025b. Early diagnosis of autism: A review of video-based motion analysis and deep learning techniques. IEEE Access. find
  31. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review, 11(12):nwae403.
  32. Cheol-Hwan Yoo, Jang-Hee Yoo, and Jaeyoon Jang. 2025. Care-vl: A domain-specialized vision-language model for early asd screening. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 57–66. Springer. find
  33. Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, and 1 others. 2025. Swift: a scalable lightweight infrastructure for fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 29733–29735. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shah-2026-rosco-omni-multimodal-llm-based/