Elamur Research⌕
☰
home / papers / 2022 / Tragic Talkers: A Shakespearean Sound- and Light-Field Dataset for Audio-Visual Machine Learning Research

Tragic Talkers: A Shakespearean Sound- and Light-Field Dataset for Audio-Visual Machine Learning Research

Paper · 2022 · Davide Berghi, Marco Volino and Philip J. B. Jackson

Summary

Immersive 3D media needs sound placed exactly where its visual source appears, but public data for this is scarce. Multi-view video collections usually ignore audio, and microphone-array corpora tend to have little or low-quality video. The authors address this gap with Tragic Talkers, a dataset in which two student actors perform scenes from "Romeo and Juliet" in Renaissance-style costume in a studio. They recorded with two identical custom devices, the AVA Rigs. Each rig holds 11 cameras and a 16-microphone array in one calibrated coordinate frame, so the sound field and the light field are captured together. Lavalier microphones and an ambisonic room microphone add clean speech. The material ranges from monologues to dialogues with movement and occlusion, giving 30 sequences seen from 22 viewpoints. Annotations include voice activity, face boxes, pose keypoints, 3D mouth positions and dialogue transcripts. The resource matters because it supports active speaker detection, speaker tracking, diarization, 3D reconstruction and object-based media research on a single coherent corpus.

Key points

  • Each AVA Rig mounts a 16-element microphone array and 11 Grasshopper3 cameras on a 0.6 m × 0.4 m perspex board. Video is 2448×2048 at 30 fps, and audio is 48 kHz, 24-bit.
  • The array is 450 mm wide but only 40 mm tall, so it localizes sound better in azimuth than in elevation, as human hearing does.
  • The dataset has 30 sequences from 22 views, or 60 single-rig sequences, giving about 3.8 hours of annotated monocular video with multi-channel audio. An earlier version had only 20 sequences with fewer labels.
  • Labels cover voice activity, 2D face bounding boxes and transcriptions; 3D mouth positions and 2D pose keypoints are provided for only a subset. The background is chroma-uniform to ease silhouette extraction.
  • The data is released at cvssp.org/data/TragicTalkers with PyTorch data loaders and feature extractors. The authors' comparison table shows it is the only such dataset combining a co-located rig, actors and a uniform background.

Interesting ideas

  • I1. In earlier work, a pretrained visual active-speaker detector produced pseudo-labels that taught a separate audio-only network to locate speakers in the frame, so one modality supervised another without manual labels.
  • I2. The scenes were scripted to climb in difficulty, from one speaker alone, to two people talking, to movement with occlusion, so each failure can be traced to a specific hard condition.
  • I3. All sensors sit on one rigid platform and share a single calibrated coordinate frame, which keeps every annotation layer aligned across modalities.
  • I4. The authors suggest splicing existing sequences into new multi-talker combinations as a cheap form of data augmentation.
  • I5. Clean close-microphone recordings are stored next to the noisy array audio as a reference, so processed signals can be checked against ground truth.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Agreed parses from the witness parsers could likewise serve as free silver supervision for a different component, such as a frame or role tagger that never sees the parsers' internals.
  • I2 (answer): Fable and test QA sets could be stratified the same way (one character, then dialogue, then interleaved actors with pronoun "occlusion") to show where reading breaks.
  • I3 (tree): Tokens, UD arcs, frames and world entities should all hang off one shared character-offset frame, so explanations stay aligned across stages.
  • I4 (world): Simple stories could be interleaved to create synthetic multi-character texts whose entity states and event order are known by construction, giving controlled tests for coreference and event ordering.
  • I5 (learn): Keeping a gold reference text alongside every noisy or converted input (for example, after UD dialect conversion) lets each repair rule be evaluated against a known target.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SurreyGuildford · United Kingdom22 works · 677 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 worksSignal Processing · 493 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 519 works

Cite

Cite as Davide Berghi, Marco Volino and Philip J. B. Jackson. 2022. Tragic Talkers: A Shakespearean Sound- and Light-Field Dataset for Audio-Visual Machine Learning Research. arXiv:2212.01892.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3565516.3565522 · arXiv 2212.01892

@misc{berghi2022tragic,
  title = {{Tragic Talkers: A Shakespearean Sound- and Light-Field Dataset for Audio-Visual Machine Learning Research}},
  author = {Davide Berghi, Marco Volino and Philip J. B. Jackson},
  year = {2022},
  doi = {10.1145/3565516.3565522},
  eprint = {2212.01892},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3565516.3565522}
}

References

36 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Triantafyllos Afouras, Yuki M. Asano, Francois Fagan, Andrea Vedaldi, and Florian Metze. 2022. Self-Supervised Object Detection From Audio-Visual Correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10575–10586. find
  2. Xavier Alameda-Pineda, Jordi Sanchez-Riera, Johannes Wienke, Vojtech Franc, Jan Cech, Kaustubh Kulkarni, Antoine Deleforge, and Radu P. Horaud. 2013. RAVEL: An Annotated Corpus for Training Robots with Audiovisual Abilities. Journal on Multimodal User Interfaces 7, 1-2 (2013), 79–91. http://hal.inria.fr/hal-00720734/en ↗
  3. R. Arandjelovic and A. Zisserman. 2017. Look, Listen and Learn. In IEEE/CVF Inter. Conf. on Computer Vision. 609–617. find
  4. Elise Arnaud, Heidi Christensen, Yan-Chen Lu, Jon Barker, Vasil Khalidov, Miles Hansard, Bertrand Holveck, Hervé Mathieu, Ramya Narasimha, Elise Taillant, Florence Forbes, and Radu Horaud. 2008. The CAVA Corpus: Synchronised Stereoscopic and Binaural Datasets with Head Movements. In Proceedings of the 10th International Conference on Multimodal Interfaces (Chania, Crete, Greece) (ICMI ’08). Association for Computing Machinery, New York, NY, USA, 109–116. https://doi.org/10.1145/1452392.1452414 ↗
  5. Davide Berghi, Adrian Hilton, and Philip J. B. Jackson. 2021. Visually Supervised Speaker Detection and Localization via Microphone Array. In IEEE 23rd Inter. Workshop on Multimedia Signal Processing. find
  6. Davide Berghi, Hanne Stenzel, Marco Volino, Adrian Hilton, and Philip J. B. Jackson. 2020. Audio-Visual Spatial Alignment Requirements of Central and Peripheral Object Events. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). 666–667. https://doi.org/10.1109/VRW50115.2020. 00184 Miguel Blanco Galindo, Philip Coleman, and Philip J. B. Jackson. 2020. Microphone array geometries for horizontal spatial audio object capture with beamforming. J. of the Audio Engineering Society 68, 5 (2020), 324–337. https://doi.org/10.17743/ jaes.2020.0025 M. Brandstein and D. Ward. 2001. Microphone Arrays: Signal Processing Techniques and Applications. Springer. https://books.google.co.uk/books?id=nND6ObXSNoEC ↗
  7. Akin Caliskan, Armin Mustafa, Evren Imre, and Adrian Hilton. 2020. Multi-view Consistency Loss for Improved Single-Image 3D Reconstruction of Clothed People. In Asian Conference on Computer Vision. Springer International Publishing, Cham, 71–88. find
  8. Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. 2019. OpenPose: Real-time Multi-Person 2D Pose Estimation using Part Affinity Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019). find
  9. Anargyros Chatzitofis, Leonidas Saroglou, Prodromos Boutis, Petros Drakoulis, Nikolaos Zioulis, Shishir Subramanyam, Bart Kevelham, Caecilia Charbonnier, Pablo Cesar, Dimitrios Zarpalas, Stefanos Kollias, and Petros Daras. 2020. HUMAN4D: A Human-Centric Multimodal Dataset for Motions and Immersive Media. IEEE Access 8 (2020), 176241–176262. https://doi.org/10.1109/ACCESS.2020.3026276 ↗
  10. Philip Coleman, Andreas Franck, Jon Francombe, Qingju Liu, Teofilo de Campos, Richard J. Hughes, Dylan Menzies, Marcos F. Simón Gálvez, Yan Tang, James Woodcock, Philip J. B. Jackson, Frank Melchior, Chris Pike, Filippo Maria Fazi, Trevor J. Cox, and Adrian Hilton. 2018. An Audio-Visual System for Object-Based Audio: From Recording to Listening. IEEE Transactions on Multimedia 20, 8 (2018), 1919–1931. https://doi.org/10.1109/TMM.2018.2794780 ↗
  11. Antoine Deleforge, Radu Horaud, Yoav Y. Schechner, and Laurent Girin. 2015. Co-Localization of Audio Sources in Images Using Binaural Features and Locally-Linear Regression. IEEE/ACM Trans. Audio, Speech and Lang. Proc. 23, 4 (apr 2015), 718–731. https://doi.org/10.1109/TASLP.2015.2405475 ↗
  12. Hoang Do, Harvey F. Silverman, and Ying Yu. 2007. A Real-Time SRP-PHAT Source Location Implementation using Stochastic Region Contraction(SRC) on a Large-Aperture Microphone Array. In IEEE Inter. Conf. on Acoustics, Speech and Signal Processing, Vol. 1. I–121–I–124. https://doi.org/10.1109/ICASSP.2007.366631 ↗
  13. Jacob Donley, Vladimir Tourbabin, Jung-Suk Lee, Mark Broyles, Hao Jiang, Jie Shen, Maja Pantic, Vamsi Krishna Ithapu, and Ravish Mehra. 2021. EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environments. (2021). arXiv:2107.04174 [cs.SD] I. D. Gebru, S. Ba, X. Li, and R. Horaud. 2018. Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion. IEEE Trans. on Pattern Analysis and Machine Intelligence 40, 5 (2018), 1086–1099. ↗
  14. R. I. Hartley and A. Zisserman. 2004. Multiple View Geometry in Computer Vision (second ed.). Cambridge University Press, ISBN: 0521540518. ↗
  15. Weipeng He, Petr Motlicek, and Jean-Marc Odobez. 2018. Deep Neural Networks for Multiple Speaker Detection and Localization. In IEEE Inter. Conf. on Robotics and Automation. 74–79. find
  16. Zhu Heming, Cao Yu, Jin Hang, Chen Weikai, Du Dong, Wang Zhangye, Cui Shuguang, and Han Xiaoguang. 2020. Deep Fashion3D: A Dataset and Benchmark for 3D Garment Reconstruction from Single Images. In Computer Vision – ECCV 2020. Springer International Publishing, 512–530. find
  17. Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. 2015. Panoptic Studio: A Massively Multiview System for Social Motion Capture. In 2015 IEEE International Conference on Computer Vision (ICCV). 3334–3342. https://doi.org/10.1109/ICCV.2015.381 ↗
  18. Guillaume Lathoud, Jean-Marc Odobez, and Daniel Gatica-Perez. 2004. AV16.3: An Audio-Visual Corpus for Speaker Localization and Tracking. In Proc. of the Machine Learning for Multimodal Interaction Workshop. 182–185. https://doi.org/10.1007/978-3-540-30568-2_16 ↗
  19. David López-Fernández, Francisco J. Madrid-Cuevas, Ángel Carmona-Poyato, Manuel J. Marín-Jiménez, and Rafael Muñoz Salinas. 2014. The AVA Multi-View Dataset for Gait Recognition. In Activity Monitoring by Multiple Distributed Sensing, Pier Luigi Mazzeo, Paolo Spagnolo, and Thomas B. Moeslund (Eds.). Springer International Publishing, 26–39. https://doi.org/10.1007/978-3-319-13323-2_3 ↗
  20. Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J. Black. 2020. Learning to Dress 3D People in Generative Clothing. In Computer Vision and Pattern Recognition (CVPR). ↗
  21. Charles Malleson, John Collomosse, and Adrian Hilton. 2020. Real-Time Multi-Person Motion Capture from Multi-View Video and IMUs. Int. J. Comput. Vision 128, 6 (jun 2020), 1594–1611. https://doi.org/10.1007/s11263-019-01270-5 ↗
  22. Mohd Azri Mohd Izhar, Marco Volino, Adrian Hilton, and Philip J. B. Jackson. 2020. Tracking Sound Sources for Object-based Spatial Audio in 3D Audio-visual Production. In Forum Acusticum. 2051–2058. https://doi.org/10.48465/fa.2020.0884 ↗
  23. Ferda Ofli, Rizwan Chaudhry, Gregorij Kurillo, René Vidal, and Ruzena Bajcsy. 2013. Berkeley MHAD: A comprehensive Multimodal Human Action Database. In 2013 IEEE Workshop on Applications of Computer Vision (WACV). 53–60. https://doi.org/ 10.1109/WACV.2013.6474999 ↗
  24. Chris Pike, Richard Taylor, Tom Parnell, and Frank Melchior. 2016. Object-based 3D Audio Production for Virtual Reality using the Audio Definition Model. Journal of the Audio Engineering Society (september 2016). find
  25. Archontis Politis, Sharath Adavanne, Daniel Krause, Antoine Deleforge, Prerak Srivastava, and Tuomas Virtanen. 2021. A Dataset of Dynamic Reverberant Sound Scenes with Directional Interferers for Sound Event Localization and Detection. In Proc. of the 6th Detection and Classification of Acoustic Scenes and Events Workshop. 125–129. ↗
  26. Albert Pumarola, Jordi Sanchez, Gary Choi, Alberto Sanfeliu, and Francesc Moreno-Noguer. 2019. 3DPeople: Modeling the Geometry of Dressed Humans. In International Conference on Computer Vision (ICCV). find
  27. X. Qian, A. Brutti, O. Lanz, M. Omologo, and A. Cavallaro. 2019. Multi-Speaker Tracking From an Audio–Visual Sensing Device. IEEE Trans. on Multimedia 21, 10 (2019), 2576–2588. ↗
  28. Iran R Roman and Juan Pablo Bello. 2021. Micarraylib: Software for Reproducible Aggregation, Standardization, and Signal Processing of Microphone Array Datasets. In Proc. of the 6th Detection and Classification of Acoustic Scenes and Events Workshop. find
  29. Florian Schweiger, Chris Pike, Tom Nixon, Matt Firth, Bruce Weir, Paul Golds, Marco Volino, Craig Cieciura, Mohd Izhar, Nick Graham-Rack, Philip J. B. Jackson, and Alex Ang. 2022. Tools for 6-DoF Immersive Audio-visual Content Capture and Production. In IBC. https://www.bbc.co.uk/rd/publications/tools-six-depth-of-field-immersive-audiovisual-content-capture-production ↗
  30. Hanne Stenzel, Davide Berghi, Marco Volino, and Philip J.B. Jackson. 2021. Naturalistic audio-visual volumetric sequences dataset of sounding actions for six degree-of-freedom interaction. In 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). 637–638. https://doi.org/10.1109/VRW52623.2021. 00201 Hanne Stenzel and Philip J. B. Jackson. 2018. Perceptual thresholds of audio-visual spatial coherence for a variety of audio-visual objects. In Conf. on Audio for Virtual and Augmented Reality. ↗
  31. T. Strybel and K. Fujimoto. 2000. Minimum audible angles in the horizontal and vertical planes: Effects of stimulus onset asynchrony and burst duration. The J. of the Acoustical Society of America 108 6 (2000), 3092–5. ↗
  32. Garvita Tiwari, Bharat Lal Bhatnagar, Tony Tung, and Gerard Pons-Moll. 2020. SIZER: A Dataset and Model for Parsing 3D Clothing and Learning Size Sensitive 3D Clothing. In European Conference on Computer Vision (ECCV). Springer. find
  33. Matt Trumble, Andrew Gilbert, Charles Malleson, Adrian Hilton, and John Collomosse. 2017. Total Capture: 3D Human Pose Estimation Fusing Video and Inertial Sensors. In 2017 British Machine Vision Conference (BMVC). find
  34. Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. 2020. Semantic Object Prediction and Spatial Sound Super-Resolution with Binaural Sounds. In The European Conf. on Computer Vision. 638–655. https://doi.org/10.1007/978-3-030-58548-8_37 ↗
  35. Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and S. Li. 2017. S3FD: Single Shot Scale-Invariant Face Detector. 2017 IEEE International Conference on Computer Vision (ICCV) (2017), 192–201. find
  36. Zhengyou Zhang. 1999. Flexible camera calibration by viewing a plane from unknown orientations. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Vol. 1. 666–673 vol.1. https://doi.org/10.1109/ICCV.1999.791289 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/berghi-2022-tragic-talkers-shakespearean-sound-light/