Elamur Research⌕
☰
home / papers / 2026 / Visual and Memory–Augmented Soccer Commentary Generation

Visual and Memory–Augmented Soccer Commentary Generation

Paper · 2026 · Haoran Sun, Natthawut Kertkeidkachorn and Kiyoaki Shirai

Summary

Soccer commentary generation models are usually trained on short clips paired with a single brief live-text sentence of about 24 words. That supervision misses much of what is visible in the video, and it treats each event in isolation, with no link to earlier plays. The authors address both gaps with two manually verified benchmarks built on SoccerNet. SN-Short enriches SoccerNet-Caption captions with cleaned, LLaMA3-consolidated content from SoccerNet-Echoes transcripts. SN-Long adds tactical, context-aware commentary that ties each target event to related earlier events in the same half. They then train an augmentation pipeline that expands incomplete captions into richer, standardized text, producing the larger MatchText corpus. Finally, they propose MatchAware, a generator that retrieves relevant earlier visual events from the match as memory. MatchAware outperforms prior baselines, including MatchVoice, on the new benchmarks. In human evaluation it gives notably more complete and deeper commentary. The work matters because it moves automatic commentary from isolated, moment-level captions toward narration that is coherent and aware of the match context.

Key points

  • SN-Short: 2,777 video-text pairs from 47 games, averaging about 35 words per event. It was built by aligning SoccerNet-Echoes transcripts within a ±15 s window of SoccerNet-Caption events. Three soccer fans with 10+ years of experience verified the annotations.
  • SN-Long: 1,765 pairs, each linked to an average of 2.84 prior events (5,006 contextual segments in total), averaging about 58 words. It was written with LLaMA3 guided by 17 summary paradigms drawn from real human commentary.
  • MatchText: an automatically augmented dataset of about 27k pairs from 424 games, averaging about 35 words per event.
  • Retrieval results: ResNet(5) visual features gave the best retrieval of historical events in most settings (R@10 = 89.32). An ablation on SN-Long showed that adding retrieval improved every metric for all visual features tested.
  • Human evaluation (97 pairs, MatchAware vs MatchVoice):
    • Completeness: 3.76 vs 2.87 (p = 0.012).
    • Depth: 3.74 vs 2.77 (p = 0.008).
    • Accuracy: 3.44 vs 3.27, a difference that was not significant (p = 0.39).
  • Limitations:
    • There is no player tracking, so the model can get player names wrong.
    • Retrieval is limited to a single match half.
    • The amount of manually annotated data is still small.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Japan Advanced Institute of Science and TechnologyNomi Shi · Japan7 works · 980 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Haoran Sun, Natthawut Kertkeidkachorn and Kiyoaki Shirai. 2026. Visual and Memory–Augmented Soccer Commentary Generation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.485

@inproceedings{sun2026visual,
  title = {{Visual and Memory–Augmented Soccer Commentary Generation}},
  author = {Haoran Sun, Natthawut Kertkeidkachorn and Kiyoaki Shirai},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.485},
  url = {https://doi.org/10.18653/v1/2026.acl-long.485}
}

References

36 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Floriane Magera Silvio Giancola Olivier Barnich Bernard Ghanem Marc Van Droogenbroeck Anthony Cioppa, Adrien Deliège. 2021. Camera calibration and player localization in soccernet-v2 and investigation of their representations for action spotting. In Proceedings of the IEEE Conference on Computer Vis… find
  2. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72.
  3. Zesen Cheng Zhiqiang Hu Yuqian Yuan Guanzheng Chen Sicong Leng Yuming Jiang Hang Zhang Xin Li Peng Jin Wenqi Zhang Fan Wang Lidong Bing Deli Zhao Boqiang Zhang, Kehan Li. 2025. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106. — [arXiv:2501.13106](https://arxiv.org/abs/2501.13106) find
  4. Alec Cook and Oktay Karakuş. 2024. Llm-commentator: Novel fine-tuning strategies of large language models for automatic commentary generation using football event data. Knowledge-Based Systems, 300:112219. find
  5. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  6. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. find
  7. Sushant Gautam, Mehdi Houshmand Sarkhoosh, Jan Held, Cise Midoglu, Anthony Cioppa, Silvio Giancola, Vajira Thambawita, Michael A. Riegler, Pal Halvorsen, and Mubarak Shah. 2024. Soccernet-echoes: A soccer game audio commentary dataset. In 2024 International Symposium on Multimedia (ISM), pages 71–78… find
  8. Silvio Giancola, Mohieddine Amine, Tarek Dghaily, and Bernard Ghanem. 2018. Soccernet: A scalable dataset for action spotting in soccer videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1711– 1721. find
  9. Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. 2024. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. CVPR. find
  10. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770– 778.
  11. Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. 2023. Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory. In Proceedings of the IEEE/CVF conference on computer vision and pat… find
  12. Byeong Jo Kim and Yong Suk Choi. 2020. Automatic baseball commentary generation using deep learning. In Proceedings of the 35th Annual ACM Symposium on Applied Computing, SAC ’20, page 1056–1065, New York, NY, USA. Association for Computing Machinery. find
  13. Tadashi Kumano, Manon Ichiki, Kiyoshi Kurihara, Hiroyuki Kaneko, Tomoyasu Komori, Toshihiro Shimizu, Nobumasa Seiyama, Atsushi Imai, Hideki Sumiyoshi, and Tohru Takagi. 2019. Generation of automated sports commentary from live sports data. In 2019 IEEE International Symposium on Broadband Multimedia… find
  14. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
  15. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and Sebastian Riedel. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Conference…
  16. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the International Conference on Machine Learning, pages 19730– 19742. PMLR.
  17. Xiang Li, Yangfan He, Shuaishuai Zu, Zhengyang Li, Tianyu Shi, Yiting Xie, and Kevin Zhang. 2025. Multi-modal large language model with rag strategies in soccer commentary generation. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6197–6206. find
  18. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81.
  19. Hassan Mkhallati, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck. 2023. Soccernet-caption: Dense video captioning for soccer broadcasts commentaries. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 5074–5085. find
  20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Association for Computational Linguistics, pages 311–318.
  21. Ji Qi, Jifan Yu, Teng Tu, Kunyu Gao, Yifan Xu, Xinyu Guan, Xiaozhi Wang, Bin Xu, Lei Hou, Juanzi Li, and 1 others. 2023. Goal: A challenging knowledge-grounded video captioning benchmark for real-time soccer commentary generation. In Proceedings of the 32nd ACM International Conference on Informatio… find
  22. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Lea…
  23. Jiayuan Rao, Zifeng Li, Haoning Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2025a. Multi-agent system for comprehensive soccer understanding. In ACM Multimedia 2025. find
  24. Jiayuan Rao, Haoning Wu, Hao Jiang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2025b. Towards universal soccer video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  25. Jiayuan Rao, Haoning Wu, Chang Liu, Yanfeng Wang, and Weidi Xie. 2024. Matchtime: Towards automatic soccer game commentary generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processinng. find
  26. Aleksander Sadikov, Martin Možina, Matej Guid, Jana Krivec, and Ivan Bratko. 2006. Automated chess tutor. In International Conference on Computers and Games, pages 13–25. Springer. find
  27. Julia Georgieva Johsan Billingham Andreas Serner Kerry Peek Bernard Ghanem Marc Van Droogenbroeck Silvio Giancola, Anthony Cioppa. 2021. Towards active learning for action spotting in association football videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Worksh… find
  28. Yasufumi Taniguchi, Yukun Feng, Hiroya Takamura, and Manabu Okumura. 2019. Generating live soccer-match commentary from play data. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AA… find
  29. Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the International Conference on Computer Vision, pages 4489–4497. find
  30. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4566–4575.
  31. You are a professional sports commentator. Your task is to expand the given event description into a more informative commentary by appending relevant contextual details from the surrounding transcript. Here is the event description (anchor): [ANCHOR_DESCRIPTION] find
  32. Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80–83. find
  33. Here is the transcript of the surrounding audio (15 seconds before and after the event): [AUDIO_TRANSCRIPT] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. 2023. Retrieval-augmented multimodal language modeling.… find
  34. Your output should preserve the wording and structure of the anchor as much as possible. Then, append natural and factually coherent details from the transcript to enrich the context. The result should read like a smooth and realistic commentary. find
  35. Ling You, Wenxuan Huang, Xinni Xie, Xiangyi Wei, Bangyan Li, Shaohui Lin, Yang Li, and Changbo Wang. 2025. Timesoccer: An end-to-end multimodal large language model for soccer commentary generation. arXiv preprint arXiv:2504.17365. — [arXiv:2504.17365](https://arxiv.org/abs/2504.17365) find
  36. Xin Zhou, Le Kang, Zhiyu Cheng, Bo He, and Jingyu Xin. 2021. Feature combination meets attention: Baidu soccer embeddings and transformer based temporal detection. arXiv preprint arXiv:2106.14447. — [arXiv:2106.14447](https://arxiv.org/abs/2106.14447) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-visual-memory-augmented-soccer-commentary/