Elamur Research⌕
☰
home / papers / 2026 / Towards Robust Evaluation of Visual Activity Recognition: Resolving Verb Ambiguity with Sense Clustering

Towards Robust Evaluation of Visual Activity Recognition: Resolving Verb Ambiguity with Sense Clustering

Paper · 2026 · Louie Hong Yao, Nicholas Jarvis and Tianyu Jiang

Summary

Exact-match scoring in visual activity recognition gives credit only when a model outputs the single gold verb. This penalizes answers that are actually reasonable. The authors identify two sources of such unfair errors: synonymy (e.g., teaching vs. lecturing) and multiple valid viewpoints on one scene (e.g., marching vs. performing).

To address this, they build verb sense clusters for the imSitu benchmark:

  • Multimodal LLMs list every fitting imSitu verb for each image.
  • Each <image,verb> pair is encoded jointly by a Llama vision model.
  • A two-stage procedure first splits each verb into fine-grained senses, then merges senses across verbs.

Model predictions are counted as correct if they fall into a cluster linked to the image. On average, an image corresponds to about four clusters of roughly two synonymous verbs each. Evaluated this way, both supervised systems and zero-shot multimodal LLMs score higher than under exact match, and the scores agree better with human judgments. The authors also show that CLIPScore barely reacts to verb changes, so it cannot replace this approach. The framework offers a reusable, more faithful evaluation that could be ported to other activity datasets.

Key points

  • Data: imSitu contains 126,102 images labelled with 504 verbs. Candidate verbs per image come from GPT-4o mini and Llama-3.2-90B, plus the original gold verb, filtered to imSitu's vocabulary.
  • Embeddings: Llama-3.2-11B encodes each image-verb pair; the 4096-dimensional representation is taken from the last token after cross-attention.
  • Clustering:
    • Both stages use K-Means or HAC (complete linkage, cosine distance), with the number of clusters chosen by Silhouette score.
    • Step 1 searches k from 2 to 16 for each verb.
    • Step 2 sets the cluster count to 504×r, with r ranging from 0.6 to 1.6.
  • CLIPScore comparison:
    • Verb-only CLIPScore retrieval reaches only about 25% Top-1 accuracy and stays under 50% Top-5.
    • Swapping the verb in a caption for an implausible or random one hardly changes the score; caption-to-caption similarity remains above 92%.
  • Code: released at github.com/ruyi101/multimodal-verb-senses.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CincinnatiCincinnati · United States6 works · 38,342 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Louie Hong Yao, Nicholas Jarvis and Tianyu Jiang. 2026. Towards Robust Evaluation of Visual Activity Recognition: Resolving Verb Ambiguity with Sense Clustering. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.32

@inproceedings{yao2026robust,
  title = {{Towards Robust Evaluation of Visual Activity Recognition: Resolving Verb Ambiguity with Sense Clustering}},
  author = {Louie Hong Yao, Nicholas Jarvis and Tianyu Jiang},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.32},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.32}
}

References

31 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization.
  2. Tadeusz Caliński and Jerzy Harabasz. 1974. A dendrite method for cluster analysis. Communications in Statistics-theory and Methods, 3(1):1–27. find
  3. Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng. 2015. Hico: A benchmark for recognizing human-object interactions in images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV 2015). find
  4. Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024. MLLM-as-ajudge: Assessing multimodal LLM-as-a-judge with vision-language benchmark. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024)… find
  5. Junhyeong Cho, Youngseok Yoon, and Suha Kwak. 2022. Collaborative transformers for grounded situation recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022). find
  6. Thilini Cooray, Ngai-Man Cheung, and Wei Lu. 2020. Attention-based context aware reasoning for situation recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020). find
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  8. Marcos Garcia. 2021. Exploring the representation of word meanings in context: A case study on homonymy and synonymy. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL 2021). find
  9. Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. 2018. AVA: A video dataset of spatio-temporally localized atomic visual actions. In 2018 IEEE/CVF Conference on Computer Vision and… find
  10. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016).
  11. Lisa Anne Hendricks and Aida Nematzadeh. 2021. Probing image-language transformers for verb understanding. In Findings of the Association for Computational Linguistics: ACL 2021 (Findings of ACL-IJCNLP 2021). find
  12. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021).
  13. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. GPT-4o system card. arXiv preprint arXiv:2410.21276.
  14. Tianyu Jiang and Ellen Riloff. 2023. Exploiting commonsense knowledge about objects for visual activity recognition. In Findings of the Association for Computational Linguistics: ACL 2023 (Findings of ACL 2023). find
  15. Dieu Thu Le, Raffaella Bernardi, and Jasper Uijlings. 2013. Exploiting language models to recognize unseen actions. In Proceedings of the 3rd ACM conference on International conference on multimedia retrieval (ICMR 2013). find
  16. Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Trung Bui, and Kyomin Jung. 2021. UMIC: An unreferenced metric for image captioning via contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Nat… find
  17. Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. 2022. Clip-event: Connecting text and images with event structures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022). find
  18. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41.
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (ACL 2002).
  20. Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. 2020. Grounded situation recognition. In Computer Vision–ECCV 2020: 16th European Conference (ECCV 2020). find
  21. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML 2021).
  22. Alessandro Raganato, Iacer Calixto, Asahi Ushio, Jose Camacho-Collados, and Mohammad Taher Pilehvar. 2023. Semeval-2023 task 1: Visual word sense disambiguation. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023). find
  23. Matteo Ruggero Ronchi and Pietro Perona. 2015. Describing common human visual actions in images. In Proceedings of the British Machine Vision Conference (BMVC 2015). find
  24. Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53–65. find
  25. Debaditya Roy, Dhruv Verma, and Basura Fernando. 2024. Clipsitu: Effectively leveraging clip for conditional predictions in situation recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2024). find
  26. Arka Sadhu, Tanmay Gupta, Mark Yatskar, Ram Nevatia, and Aniruddha Kembhavi. 2021. Visual semantic role labeling for video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021). find
  27. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2024. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. arXiv preprint arXiv:2406.12624. find
  28. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015).
  29. Xiao Wang, Zongzhen Wu, Bo Jiang, Zhimin Bao, Lin Zhu, Guoqi Li, Yaowei Wang, and Yonghong Tian. 2024. HARDVS: Revisiting human activity recognition with dynamic vision sensors. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2024). find
  30. Bangpeng Yao, Xiaoye Jiang, Aditya Khosla, Andy Lai Lin, Leonidas Guibas, and Li Fei-Fei. 2011. Human action recognition by learning bases of action attributes and parts. In 2011 International Conference on Computer Vision (ICCV 2011). find
  31. Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual semantic role labeling for image understanding. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yao-2026-robust-evaluation-visual-activity-recognition/