Elamur Research⌕
☰
home / papers / 2022 / Model Transfer for Event tracking as Transcript Understanding for Videos of Small Group Interaction

Model Transfer for Event tracking as Transcript Understanding for Videos of Small Group Interaction

Summary

The authors approach understanding videos of small-group work as something broader than analyzing the conversation transcript. They see tracking each participant throughout the entire interaction as the foundation for this. In close collaborative work this is difficult: people sit next to each other, move in short bursts, and often leave the frame and come back. The authors take the online tracker DeepSORT, trained to follow pedestrians on the street, and transfer it to videos in which two people jointly assemble a robot. First, the YOLO detector is replaced with Detectron, yielding the DeepDSORT version. Then they add three changes that account for the fixed composition of the group: tracks are no longer deleted because of their "age", the number of tracks cannot exceed the number of participants, and the appearance feature is smoothed over time. The modified DeepDSORT+ model outperforms a strong baseline tracker. Replacing the detector alone already gives a noticeable gain, and each of the three additions improves the result further. An ablation study and a qualitative analysis confirm this. The work serves as a proof of concept for multimodal learning analytics and for supporting group work.

Key points

  • The base is DeepSORT, which combines a Kalman filter and the Mahalanobis distance with a threshold of 0.95 with a CNN-based REID model using cosine distance with a threshold of 0.2. The YOLO detector is replaced with Detectron.
  • The appearance feature of each track is updated as a moving average with α = 0.1, so most of the weight falls on previous observations.
  • In the ablation, smoothing reduces the number of ID switches and false positives. Without the track limit, false positives increase. If max age is set back to 100 frames, false positives rise sharply and IDF1 drops.
  • Limitations: the approach is designed for a closed environment with a fixed number of participants. It can mistakenly take people in the background into account, and similar clothing among participants confuses REID. The authors propose taking scene depth into account and moving to more sophisticated REID models.
  • The paper was published as part of the First Workshop on Transcript Understanding (COLING 2022) and was partially funded by the NSF (grants 2100401 and 1917955).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{agarwal2022model,
  title = {{Model Transfer for Event tracking as Transcript Understanding for Videos of Small Group Interaction}},
  author = {Agarwal, Sumit and Vitiello, Rosanna and Rosé, Carolyn},
  booktitle = {Proceedings of the First Workshop On Transcript Understanding},
  year = {2022},
  doi = {10.48448/7w8j-hh28},
  url = {https://doi.org/10.48448/7w8j-hh28}
}

References

18 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Karan Ahuja, Dohyun Kim, Franceska Xhakaj, Virag Varga, Anne Xie, Stanley Zhang, Jay Eric Townsend, Chris Harrison, Amy Ogan, and Yuvraj Agarwal. 2019. Edusense: Practical classroom sensing at scale. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 3(3). find
  2. Joseph Redmon and Ali Farhadi. 2018. Yolov3: An incremental improvement. ArXiv, abs/1804.02767. — [arXiv:1804.02767](https://arxiv.org/abs/1804.02767) find
  3. Karan Ahuja, Deval Shah, Sujeath Pareddy, Franceska Xhakaj, Amy Ogan, Yuvraj Agarwal, and Chris Harrison. 2021. Classroom digital twins with instrumentation-free gaze tracking. CHI ’21, New York, NY, USA. Association for Computing Machinery. find
  4. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc. find our card
  5. Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. 2016. Simple online and realtime tracking. In 2016 IEEE International Conference on Image Processing (ICIP), pages 3464–3468. find
  6. Carolyn Rosé, Yi-Chia Wang, Yue Cui, Jaime Arguello, Karsten Stegmann, Armin Weinberger, and Frank Fischer. 2008. Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. I. J. Computer-Supported Col… find
  7. Mauro Cherubini, Marc-Antoine Nüssli, and Pierre Dil-lenbourg. 2008. Deixis and gaze in collaborative work at a distance (over a shared map): A computational model to detect misunderstandings. In Proceedings of the 2008 Symposium on Eye Tracking Research Applications, ETRA ’08, page 173–180, New Yo… find
  8. Daniel Spikol, Emanuele Ruffaldi, Giacomo Dabisias, and Mutlu Cukurova. 2018. Supervised machine learning in multimodal learning analytics for estimating success in project-based learning. Journal of computer assisted learning., 34(4). find
  9. P. Dendorfer, H. Rezatofighi, A. Milan, J. Shi, D. Cremers, I. Reid, S. Roth, K. Schindler, and L. Leal-Taixé. 2020. Mot20: A benchmark for multi object tracking in crowded scenes. arXiv:2003.09003[cs]. ArXiv: 2003.09003. — [arXiv:2003.09003](https://arxiv.org/abs/2003.09003) find
  10. Hideaki Uchiyama and Eric Marchand. 2012. Object Detection and Pose Tracking for Augmented Reality: Recent Approaches. In 18th Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV), Kawasaki, Japan. find
  11. Caglar Genc and Jonna Häkkilä. 2021. Using body tracking for involving museum visitors in digital storytelling. In Augmented Humans Conference 2021, AHs’21, page 304–306, New York, NY, USA. Association for Computing Machinery. find
  12. Chunyu Wang, Yizhou Wang, and Alan L. Yuille. 2013. An approach to pose-based action recognition. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pages 915–922. find
  13. Rohit Kumar, Carolyn P. Rosé, Yi-Chia Wang, Mahesh Joshi, and Allen Robinson. 2007. Tutorial dialogue as adaptive collaborative learning support. In Proceedings of the 2007 Conference on Artificial Intelligence in Education: Building Technology Rich Learning Contexts That Work, page 383–390, NLD. I… find
  14. Nicolai Wojke, Alex Bewley, and Dietrich Paulus. 2017. Simple online and realtime tracking with a deep association metric. In 2017 IEEE International Conference on Image Processing (ICIP), pages 3645–3649. IEEE. find
  15. Chenxu Luo, Chang Ma, Chunyu Wang, and Yizhou Wang. 2017. Learning discriminative activated sim-plices for action recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1). https://github.com/facebookresearch/ detectron2 . ↗
  16. Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. 2019. Detectron2. find
  17. Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, and Wenyu Liu. 2021. Fairmot: On the fairness of detection and re-identification in multiple object tracking. International Journal of Computer Vision, 129:3069–3087. find
  18. Bruce M. McLaren, Oliver Scheuer, Maarten De Laat, Rakheli Hever, Reuma De Groot, and Carolyn P. Rosé. 2007. Using machine learning techniques Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. 2015. Scalable person re-identification: A benchmark. In Computer Vision, IEEE… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/agarwal-2022-model-transfer-event-tracking-as/