Elamur Research⌕
☰
home / papers / 2026 / InsAT: Instance-aware Semantic Alignment and Transfer from Human–Object Keypoints for Zero-to-Few-shot Action Understanding

InsAT: Instance-aware Semantic Alignment and Transfer from Human–Object Keypoints for Zero-to-Few-shot Action Understanding

Paper · 2026 · Kazuki Tsutsukawa

Summary

Recognizing actions from skeletal keypoints is attractive because it ignores appearance and protects privacy. However, zero-shot (ZS) keypoint methods have mostly modeled body motion alone and rarely use the objects people handle. Moving such models from ZS to few-shot use has also received little study.

The paper presents InsAT, one unified approach that handles both ZS recognition and adaptation from zero to few shots (Z2F). Human and object keypoints are encoded into per-instance features. Offline, GPT-4o writes, for each class label, an action summary plus separate descriptions of the person, the objects and their interaction. Instance-level contrastive learning links each text to the matching visual instances, so interaction knowledge carries over from seen to unseen classes.

For Z2F, Instance-level Visual Adaptation (IVA) builds visual prototypes from a few labeled examples and mixes them with text similarity, keeping both encoders frozen. An open-vocabulary detector lets new object types be added without retraining.

InsAT outperforms earlier keypoint-based ZS methods and is competitive with large vision–language models. This makes it a privacy-friendly, data-efficient route to action understanding.

Key points

  • Inputs: Human keypoints come from OpenPose or HRNet. Object keypoints come from COCO-trained PPNv2 (90 categories) or D-FINE (80), with Grounding DINO for broader coverage. Each object is represented by 8 points, and up to two humans and two objects are kept per frame.
  • Kinetics-ObjShared subset: The authors built this subset of different actions sharing the same object (12 seen / 10 unseen classes). Instance-level descriptions improve ZS accuracy by 5.38 points over plain label text, a larger gain than on other Kinetics-400/200 splits.
  • Caption granularity: Per-video (data-level) GPT-4o captions for 823 training videos raise accuracy only slightly over class-level captions (51.44% → 52.83%). Template-based captions and an open-source LLM also work.
  • Robustness to detection noise (80% of object instances corrupted): InsAT loses only 2.36 points with missing objects, versus 6.44 for the model without InsAT. It suffers more from swapped object categories (−15.49 vs −11.51), reflecting its reliance on category semantics.
  • Limitations: The method depends on reliable keypoint extraction and may degrade with crowds, occlusion or small objects. The authors also note possible misuse for surveillance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksHuman-Computer Interaction · 141 works

Cite

Cite as Kazuki Tsutsukawa. 2026. InsAT: Instance-aware Semantic Alignment and Transfer from Human–Object Keypoints for Zero-to-Few-shot Action Understanding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1690

@inproceedings{tsutsukawa2026insat,
  title = {{InsAT: Instance-aware Semantic Alignment and Transfer from Human–Object Keypoints for Zero-to-Few-shot Action Understanding}},
  author = {Kazuki Tsutsukawa},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1690},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1690}
}

References

48 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shahzad Ahmad, Sukalpa Chanda, and Yogesh S Rawat. 2025. T2L: Efficient zero-shot action recognition with temporal token learning. Transactions on Machine Learning Research, 2025. find
  2. Salman Ali, Shaheer Athar, Hamed Larijani, Josef Kittler, and Henglin Dai. 2019. Skeleton based zero shot action recognition in joint pose-language semantic space. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1459–1467. IEEE. find
  3. Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1302–1310. find
  4. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-End Object Detection With Transformers. In Proceedings of the European Conference on Computer Vision (ECCV).
  5. João Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? A new model and the kinetics dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724–4733. IEEE. find
  6. Yang Chen, Jingcai Guo, Song Guo, and Dacheng Tao. 2025. Neuron: Learning context-aware evolving representations for zero-shot skeleton action recognition. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8721–8730. find
  7. Ming Cheng, Kunjing Cai, and Ming Li. 2021. Rwf-2000: An open large scale video database for violence detection. In Proceedings of the 25th International Conference on Pattern Recognition (ICPR), pages 4183–4190. IEEE. find
  8. Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, and Bo Dai. 2022. Revisiting skeleton-based action recognition. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2959–2968. find
  9. Andrea Frome, Greg S. Corrado, Jon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, and Tomás Mikolov. 2013. DeViSE: A Deep Visual-Semantic Embedding Model. In Advances in Neural Information Processing Systems (NeurIPS). find
  10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  11. Pranay Gupta, Divyanshu Sharma, and Ravi Kiran Sar-vadevabhatla. 2021. Syntactically Guided Generative Embeddings for Zero-Shot Skeleton Action Recognition. In Proceedings of the IEEE International Conference on Image Processing (ICIP). find
  12. Ryo Hachiuma, Fumiaki Sato, and Taiki Sekii. 2023. Unified keypoint-based action recognition framework via structured keypoint pooling. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22962–22971. find
  13. Tal Hassner, Yossi Itcher, and Orit Kliper-Gross. 2012. Violent flows: Real-time detection of violent crowd behavior. In 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages 1–6. find
  14. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778.
  15. Stephanie Käs, Anton Burenko, Louis Markert, Onur Alp Culha, Dennis Mack, Timm Linder, and Bastian Leibe. 2025. How do foundation models compare to skeleton-based approaches for gesture recognition in human-robot interaction? Preprint, arXiv:2506.20795. — [arXiv:2506.20795](https://arxiv.org/abs/2506.20795) find
  16. Hilde Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. 2011. HMDB: A large video database for human motion recognition. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). find
  17. Sang Uk Lee, Andreas Hofmann, and Brian Williams. 2019. A Model-Based Human Activity Recognition for Human–Robot Collaboration. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS). find
  18. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV).
  19. Hongda Liu, Yunfan Liu, Min Ren, Hao Wang, Yunlong Wang, and Zhenan Sun. 2025a. Revealing key details to see differences: A novel prototypical perspective for skeleton-based action recognition. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 29248–29257. find
  20. Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C. Kot. 2020a. NTU RGB+D 120: A large-scale benchmark for 3d human activity understanding. IEEE Trans. Pattern Anal. Mach. Intell., 42(10):2684–2701. find
  21. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. 2025b. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In Proceedings of the European Conference on Computer Vision (E…
  22. Xingyu Liu, Sanping Zhou, Le Wang, and Gang Hua. 2023. Parallel attention interaction network for few-shot skeleton-based action recognition. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1379–1388. find
  23. Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang. 2020b. Disentangling and unifying graph convolutions for skeleton-based action recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 140–149. find
  24. Ning Ma, Hongyi Zhang, Xuhui Li, Sheng Zhou, Zhen Zhang, Jun Wen, Haifeng Li, Jingjun Gu, and Jiajun Bu. 2022. Learning spatial-preserved skeleton representations fornbsp;few-shot action recognition. In Proceedings of the European Conference on Computer Vision (ECCV). find
  25. Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. 2022. Expanding language-image pretrained models fornbsp;general video recognition. In Proceedings of the European Conference on Computer Vision (ECCV). find
  26. OpenAI. 2024a. Gpt-4o system card. OpenAI Technical Report. Reported: 2024-8-8.
  27. OpenAI. 2024b. New embedding models and API updates. Reported: 2024-02-14. find
  28. Yansong Peng, Hebei Li, Peixi Wu, Yueyi Zhang, Xiaoyan Sun, and Feng Wu. 2024. D-FINE: Redefine regression task in detrs as fine-grained distribution refinement. arXiv preprint arXiv:2410.13842. find
  29. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th Internatio…
  30. Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023. Fine-tuned clip models are efficient video learners. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6545–6554. find
  31. Taiki Sekii. 2018. Pose proposal networks. In Proceedings of the European Conference on Computer Vision (ECCV). find
  32. Taiki Sekii. 2021. Object Detection Method and Object Detection Device. Patent WO2021/117363. find
  33. Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. 2016. NTU RGB+D: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1010–1019. IEEE. find
  34. Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. 2020. Finegym: A hierarchical video dataset for fine-grained action understanding. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2613–2622. find
  35. Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. 2020. Decoupled spatial-temporal attention network for skeleton-based action-gesture recognition. In Proceedings of the Asian Conference on Computer Vision (ACCV), pages 38–53. find
  36. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. MPNet: Masked and permuted pre-training for language understanding. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS).
  37. Yi-Fan Song, Zhang Zhang, Caifeng Shan, and Liang Wang. 2023. Constructing stronger and faster baselines for skeleton-based action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1474–1488. find
  38. Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. 2019. Deep high-resolution representation learning for human pose estimation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5686–5696. find
  39. Mengmeng Wang, Jiazheng Xing, and Yong Liu. 2021. Actionclip: A new paradigm for video action recognition. Preprint, arXiv:2109.08472. — [arXiv:2109.08472](https://arxiv.org/abs/2109.08472) find
  40. Michael Wray, Gabriela Csurka, Diane Larlus, and Dima Damen. 2019. Fine-grained action retrieval through multiple parts-of-speech embeddings. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 450–459. find
  41. Yongqin Xian, Bernt Schiele, and Zeynep Akata. 2017. Zero-shot learning — the good, the bad and the ugly. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3077–3086. find
  42. Wangmeng Xiang, Chao Li, Yuxuan Zhou, Biao Wang, and Lei Zhang. 2023. Generative action description prompts for skeleton-based action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10242– 10251. find
  43. Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. 2021. Video-CLIP: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processi… find
  44. Sijie Yan, Yuanjun Xiong, and Dahua Lin. 2018. Spatial temporal graph convolutional networks for skeleton-based action recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). find
  45. Yujie Zhou, Wenwen Qiang, Anyi Rao, Ning Lin, Bing Su, and Jiaqi Wang. 2023. Zero-Shot Skeleton-Based Action Recognition via Mutual Information Estimation and Maximization. In Proceedings of the 31st ACM International Conference on Multimedia. find
  46. Anqi Zhu, Qiuhong Ke, Mingming Gong, and James Bailey. 2024. Part-aware unified representation of language and skeleton for zero-shot action recognition. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18761– 18770. find
  47. Anqi Zhu, Jingmin Zhu, James Bailey, Mingming Gong, and Qiuhong Ke. 2025a. Semantic-guided cross-modal prompt learning for skeleton-based zero-shot action recognition. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13876–13885. find
  48. Jingmin Zhu, Anqi Zhu, Hossein Rahmani, Jun Liu, Mohammed Bennamoun, and Qiuhong Ke. 2025b. Boosting skeleton-based zero-shot action recognition with training-free test-time adaptation. In The 39th Annual Conference on Neural Information Processing Systems (NeurIPS). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tsutsukawa-2026-insat-instance-aware-semantic-alignment/