Elamur Research⌕
☰
home / papers / 2026 / A2O: LLM-based Agentic Learning of Action-to-Object Features for Video Action Recognition

A2O: LLM-based Agentic Learning of Action-to-Object Features for Video Action Recognition

Paper · 2026 · Taiki Sekii and Fumiaki Sato

Summary

Video action recognizers built on vision–language pretraining or self-supervised foundation models often rely on shortcuts, such as scene backgrounds or objects that merely co-occur with an action. Object-centric recognizers avoid these shortcuts but discard context, which limits accuracy. Sekii and Sato (CyberAgent) propose A2O, which combines the two approaches.

An LLM agent uses its commonsense knowledge of human actions to pick object classes that matter for the target actions. It also picks nontarget classes that capture irrelevant cues so they can be downweighted. An open-vocabulary object detector (OVD) finds these classes in every frame. An adapter then fuses the detections with the raw video and feeds the resulting object-centric features into a video foundation model such as V-JEPA2 during supervised fine-tuning. This keeps the full input while steering where the model attends. The class set is refined through in-context reinforcement learning (ICRL), with validation scores from earlier trials serving as feedback.

On public benchmarks, A2O yields modest but statistically significant gains over V-JEPA2. This suggests LLM knowledge can guide how the backbone learns features, not only act as a classifier head.

Key points

  • Evaluation datasets are SSv2, MultiSports and Kinetics-400. The recognizer is V-JEPA2, the object detector is an OVD, and the LLM agent proposes up to C_max object classes.
  • In repeated runs with matched seeds, A2O beat V-JEPA2 on all three reported benchmarks:
    • 75.4 → 76.4 (p = 0.0215)
    • 83.3 → 84.3 (p = 0.0003)
    • 85.9 → 86.8 (p = 0.0053)
  • On MultiSports, A2O was compared with a V-JEPA2 baseline enlarged with more MoE experts:
    • accuracy rose from 59.2% to 59.8%;
    • hyperparameter search time fell from 145 to 38 GPU-hours;
    • inference time fell from 1812 to 764 ms.
  • An ablation study supports ICRL's role in optimizing the composition of target and nontarget objects.
  • The authors list three limitations:
    • dependence on detector quality (occlusion, blur, small objects);
    • sensitivity to the choice of LLM and the prompt;
    • possible ICRL instability and extra training cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksHuman-Computer Interaction · 141 works

Cite

Cite as Taiki Sekii and Fumiaki Sato. 2026. A2O: LLM-based Agentic Learning of Action-to-Object Features for Video Action Recognition. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1805

@inproceedings{sekii2026a2o,
  title = {{A2O: LLM-based Agentic Learning of Action-to-Object Features for Video Action Recognition}},
  author = {Taiki Sekii and Fumiaki Sato},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1805},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1805}
}

References

46 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In SIGKDD.
  2. Hussein Altabrawee and Mohd Halim Mohd Noor. 2025. Stclr: Sparse temporal contrastive learning for video representation. Neurocomputing, 630:129694. find
  3. Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, and Bo Dai. 2022. Revisiting Skeleton-Based Action Recognition. In CVPR. find
  4. Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, and Jun Liu. 2024. Llms are good sign language translators. In CVPR. find
  5. Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyńska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. 2017. The "something something" video database for learning… find
  6. Ryo Hachiuma, Fumiaki Sato, and Taiki Sekii. 2023. Unified Keypoint-based Action Recognition Framework via Structured Keypoint Pooling. In CVPR. find
  7. De-An Huang, Vignesh Ramanathan, Dhruv Mahajan, Lorenzo Torresani, Manohar Paluri, Li Fei-Fei, and Juan Carlos Niebles. 2018. What makes a video a video: Analyzing temporal information in video understanding models and datasets. In CVPR. find
  8. Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950. find
  9. Minji Kim, Dongyoon Han, Taekyung Kim, and Bohyung Han. 2024. Leveraging temporal contextualization for video action recognition. In ECCV. find
  10. Stephanie Käs, Anton Burenko, Louis Markert, Onur Alp Culha, Dennis Mack, Timm Linder, and Bastian Leibe. 2025. How do foundation models compare to skeleton-based approaches for gesture recognition in human-robot interaction? In RO-MAN. find
  11. Yixuan Li, Lei Chen, Runyu He, Zhenzhi Wang, Gangshan Wu, and Limin Wang. 2021. Multisports: A multi-person video dataset of spatio-temporally localized sports actions. In ICCV. find
  12. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. Video-LLaVA: Learning united visual representation by alignment before projection. In EMNLP.
  13. Neelu Madan, Andreas Moegelmose, Rajat Modi, Yogesh S. Rawat, and Thomas B. Moeslund. 2024. Foundation models for video understanding: A survey. find
  14. Lingyi Meng, Lin Yang, and Enhao Zheng. 2024. Hierarchical human motion intention prediction for increasing efficacy of human-robot collaboration. IEEE Robotics and Automation Letters, 9(9):7637– 7644. find
  15. Amir Moeini, Jiuqi Wang, Jacob Beck, Ethan Blaser, Shimon Whiteson, Rohan Chandra, and Shangtong Zhang. 2025. A survey of in-context reinforcement learning. arXiv preprint arXiv:2502.07978. — [arXiv:2502.07978](https://arxiv.org/abs/2502.07978) find
  16. Haoxuan Qu, Yujun Cai, and Jun Liu. 2024. Llms are good action recognizers. In CVPR. find
  17. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In ICML.
  18. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356.
  19. Ali Rasekh, Erfan Bagheri Soula, Omid Daliran, Simon Gottschalk, and Mohsen Fayyaz. 2025. Enhancing temporal understanding in video-llms through stacked temporal attention in vision encoders. In NeurIPS. find
  20. Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. 2024. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:24… — [arXiv:2401.14159](https://arxiv.org/abs/2401.14159) find
  21. Fumiaki Sato, Ryo Hachiuma, and Taiki Sekii. 2023. Prompt-guided zero-shot anomaly action recognition using pretrained deep skeleton features. In CVPR. find
  22. Nina Shvetsova, Arsha Nagrani, Bernt Schiele, Hilde Kuehne, and Christian Rupprecht. 2025. Unbiasing through textual descriptions: Mitigating representation bias in video benchmarks. In CVPR. find
  23. Hao Song, Yangjun Ou, and Chen Wang. 2025. Cpr-clip: Cross-modal consistent and prompt-diverse regularized clip for action recognition. In MMAsia. find
  24. Federico Spurio, Emad Bahrami, Gianpiero Francesca, and Juergen Gall. 2025. Hierarchical vector quantization for unsupervised action segmentation. In AAAI. find
  25. David Steinmann, Felix Divo, Maurice Kraus, Antonia Wüst, Lukas Struppek, Felix Friedrich, and Kristian Kersting. 2025. Navigating shortcuts, spurious correlations, and confounders: From origins via detection to mitigation. find
  26. Gemini Team, Rohan Anil, Sebastian Borgeaud, and 1 others. 2025. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  27. Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. In NeurIPS. find
  28. Alexander Tornede, Difan Deng, Theresa Eimer, Joseph Giovanelli, Aditya Mohan, Tim Ruhkopf, Sarah Segel, Daphne Theodorakopoulos, Tanja Tornede, Henning Wachsmuth, and Marius Lindauer. 2024. Automl in the age of large language models: Current challenges, future opportunities and risks. TMLR. find
  29. Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. 2025. AutoML-agent: A multi-agent LLM framework for full-pipeline autoML. In ICML. find
  30. Chenting Wang, Yuhan Zhu, Yicheng Xu, Jiange Yang, Ziang Yan, Yali Wang, Yi Wang, and Limin Wang. 2025a. Internvideo-next: Towards general video foundation models without video-text supervision. arXiv preprint arXiv:2512.01342. — [arXiv:2512.01342](https://arxiv.org/abs/2512.01342) find
  31. Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. 2019. Deep high-resolution representation learning for visual recognition. PAMI. find
  32. Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. 2023. Videomae v2: Scaling video masked autoencoders with dual masking. In CVPR. find
  33. Weiyun Wang, Zhangwei Gao, Lixin Gu, and 1 others. 2025b. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.
  34. Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Chenting Wang, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, and 1 others. 2024. Internvideo2: Scaling video foundation models for multimodal video understanding. In ECCV. find
  35. Yi Wang, Xinhao Li, Ziang Yan, and 1 others. 2025c. Internvideo2.5: Empowering video mllms with long and rich context modeling. arXiv preprint arXiv:2501.12386. — [arXiv:2501.12386](https://arxiv.org/abs/2501.12386) find
  36. Philippe Weinzaepfel and Grégory Rogez. 2021. Mimetics: Towards Understanding Human Actions out of Context. IJCV, 129(5):1675–1690. find
  37. Fei Wu, Qingzhong Wang, Jiang Bian, Ning Ding, Feixiang Lu, Jun Cheng, Dejing Dou, and Haoyi Xiong. 2023. A survey on video action recognition in sports: Datasets, methods and applications. IEEE Transactions on Multimedia, 25:7943–7966. find
  38. Jinheng Xie, Jianfeng Xiang, Junliang Chen, Xianxu Hou, Xiaodong Zhao, and Linlin Shen. 2022. C2am: Contrastive learning of class-agnostic activation map for weakly supervised object localization and semantic segmentation. In CVPR. find
  39. An Yang, Anfeng Li, Baosong Yang, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  40. Haosen Yang, Deng Huang, Bin Wen, Jiannan Wu, Hongxun Yao, Yi Jiang, Xiatian Zhu, and Zehuan Yuan. 2024. Motionmae: Self-supervised video representation learning with motion-aware masked autoencoders. In BMVC. find
  41. Zuhao Yang, Yingchen Yu, Yunqing Zhao, Shijian Lu, and Song Bai. 2025b. Timeexpert: An expert-guided video llm for video temporal grounding. In ICCV. find
  42. Jiapeng Yao, Lantian Zhang, and Jiping Huang. 2025. Evaluation of large language model-driven automl in data and model management from human-centered perspective. Frontiers in Artificial Intelligence, 8:1590105. find
  43. Runpeng Yu, Weihao Yu, and Xinchao Wang. 2024. Attention prompting on image for large vision-language models. In ECCV. find
  44. Shujian Zhang, Chengyue Gong, Lemeng Wu, Xingchao Liu, and Mingyuan Zhou. 2023. Automl-gpt: Automatic machine learning with gpt. arXiv preprint arXiv:2305.02499.
  45. Long Zhao, Nitesh B. Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J. Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, Rachel Hornung, Florian Schroff, Ming-Hsuan Yang, David A. Ross, Huisheng Wang, Hartwig Adam, Mikhail Sirotenko, Ting Liu, and Boqing Gong. 2024. Video-Prism: A fou… find
  46. Fatimah Zohra, Chen Zhao, Shuming Liu, and Bernard Ghanem. 2025. Effectiveness of max-pooling for fine-tuning clip on videos. In CVPRW. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sekii-2026-a2o-llm-based-agentic-learning/