Elamur Research⌕
☰
home / papers / 2026 / Beyond Transcripts: A Renewed Perspective on Audio Chaptering

Beyond Transcripts: A Renewed Perspective on Audio Chaptering

Paper · 2026 · Fabian Retkowski, Maike Züfle, Thai Binh Nguyen et al.

Summary

Automatic chaptering of long recordings such as podcasts, lectures and videos has mostly been handled as text segmentation over transcripts. This leaves open questions about the value of audio, the effect of ASR errors, and how to evaluate fairly when transcripts differ. The authors compare three families of systems on YTSeg:

  • MiniSeg, a hierarchical text model, with and without fused hand-crafted acoustic features (pauses, speaking rate, pitch, loudness, speaker cues);
  • AudioSeg, a new transcript-free model built on Whisper encoder representations, which classifies 6-second windows inside 30-second chunks;
  • Multimodal LLMs (Qwen2.5-Omni, Qwen3-Omni), used via LoRA fine-tuning or in-context learning.

They also formalize evaluation protocols. Text-space variants (R1, H1–H3) are contrasted with transcript-invariant time-space variants: T1 uses discretized time chunks, and T2 uses collar-based F1 in continuous time. AudioSeg clearly beats the text-based systems. Among acoustic features, pauses account for almost all of the gain. MLLMs are held back by context length and weak instruction following, though they look promising on short audio. The work matters because it shows that audio can match or exceed transcripts for this task and because it offers fairer comparison tools.

Key points

  • Released resources: the chunkseg evaluation package (GitHub, PyPI), the AudioSeg model (HF), and extra annotations merged into the YTSeg repository.
  • Speaker composition: multi-speaker content is hard for every system. AudioSeg holds up best, with F1 falling from 54.62 (single speaker) to 35.64 (multi-speaker). Speaker features add only +0.67 F1 overall but +2.95 on multi-speaker videos (26.10 → 29.05).
  • Limits of transcript-based protocols: even oracle boundary selection reaches only about 73–81% F1. Coarser ASR sentence segmentation can also inflate text-space scores for identical predictions.
  • Temporal grounding in MLLMs: when asked to output timestamps without a transcript, Qwen3-Omni reaches F1=12.06 and Qwen2.5-Omni 0.09. This points to timestamp grounding, rather than segmentation itself, as the main weakness.
  • Transfer to AMI meetings: AudioSeg stays the best system on AMI, while Qwen3-Omni degrades. Pretraining on YTSeg before AMI outperforms training on AMI alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓Karlsruhe Institute of TechnologyKarlsruhe · Germany12 works · 834 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSignal Processing · 452 worksCommunication · 97 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 480 works

Cite

Cite as Fabian Retkowski, Maike Züfle, Thai Binh Nguyen, Jan Niehues and Alexander Waibel. 2026. Beyond Transcripts: A Renewed Perspective on Audio Chaptering. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.396

@inproceedings{retkowski2026beyond,
  title = {{Beyond Transcripts: A Renewed Perspective on Audio Chaptering}},
  author = {Fabian Retkowski, Maike Züfle, Thai Binh Nguyen, Jan Niehues and Alexander Waibel},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.396},
  url = {https://doi.org/10.18653/v1/2026.acl-long.396}
}

References

41 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Doug Beeferman, Adam Berger, and John Lafferty. 1999. Statistical Models for Text Segmentation. Machine Learning, 34(1):177–210. find
  2. Hervé Bredin. 2017. pyannote.metrics: A toolkit for reproducible evaluation, diagnostic, and error analysis of speaker diarization systems. In Interspeech 2017, pages 3587–3591. find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  4. Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Iain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Wellner. 2006. The ami meeting corp… find
  5. Chris Fournier. 2013. Evaluating text segmentation using boundary edit distance. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1702–1712, Sofia, Bulgaria. Association for Computational Linguistics. find
  6. Steffen Freisinger, Philipp Seeberger, Thomas Ranzen-berger, Tobias Bocklet, and Korbinian Riedhammer. 2025. Towards Multi-Level Transcript Segmentation: LoRA Fine-Tuning for Table-of-Contents Generation. In Interspeech 2025, pages 276–280. find
  7. Azin Ghazimatin, Ekaterina Garmash, Gustavo Penha, Kristen Sheets, Martin Achenbach, Oguz Se-merci, Remi Galvez, Marcus Tannenberg, Sahitya Mantravadi, Divya Narayanan, Ofeliya Kalaydzhyan, Douglas Cole, Ben Carterette, Ann Clifton, Paul N. Bennett, Claudia Hauff, and Mounia Lalmas. 2024. PODTILE: f… find
  8. Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. 2025. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. Preprint, arXiv:2507.08128. — [arXiv:2507.08128](https://arxiv.org/abs/2507.08128) find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  10. Camille Guinaudeau, Guillaume Gravier, and Pascale Sé-billot. 2010. Improving asr-based topic segmentation of tv programs with confidence measures and semantic relations. In Interspeech 2010, pages 1365–1368. find
  11. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE ACM Trans. Audio Speech Lang. Process., 29:3451–3460.
  12. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  13. Nithin Rao Koluguri, Taejin Park, and Boris Ginsburg. 2022. Titanet: Neural model for speaker representation with 1d depth-wise separable convolutions and global context. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022,… find
  14. Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant. 2018. Text Segmentation as a Supervised Learning Task. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers),…
  15. Ludwig Kürzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, and Gerhard Rigoll. 2020. Ctc-segmentation of large corpora for german end-to-end speech recognition. In Speech and Computer, pages 267–278, Cham. Springer International Publishing. find
  16. Catherine Lai, Mireia Farrús, and Johanna D. Moore. 2016. Automatic Paragraph Segmentation with Lexical and Prosodic Features. In Interspeech 2016, pages 1034–1038. ISSN: 2958-1796. find
  17. Catherine Lai, Mireia Farrús, and Johanna D. Moore. 2020. Integrating lexical and prosodic features for automatic paragraph segmentation. Speech Communication, 121:44–57. find
  18. Xinjian Li. 2023. Low-Resource Speech Recognition for Thousands of Languages. find
  19. Chen-Hsuan Liao and Jiun-Yu Wu. 2023. Learning analytics on video-viewing engagement in a flipped statistics course: Relating external video-viewing patterns to internal motivational dynamics and performance. Comput. Educ., 197(C). find
  20. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  21. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  22. Yikun Liu, Yuan Liu, Shangzhe Di, Haicheng Wang, Zhongyin Zhao, Le Tian, Xiao Zhou, Jie Zhou, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. 2026. Versavit: Enhancing mllm vision backbones via task-guided optimization. arXiv preprint arXiv:2602.09934. — [arXiv:2602.09934](https://arxiv.org/abs/2602.09934) find
  23. Michal Lukasik, Boris Dadachev, Kishore Papineni, and Gonçalo Simões. 2020. Text Segmentation by Cross Segment Attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4707–4716, Online. Association for Computational Linguistics. find
  24. Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. 2016. Metrics for polyphonic sound event detection. Applied Sciences, 6(6). find
  25. Benjamin Minixhofer, Jonas Pfeiffer, and Ivan Vulić. 2023. Where’s the point? self-supervised multilingual punctuation-agnostic sentence segmentation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7215–7235, Toronto, Canada.… find
  26. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learn…
  27. Fabian Retkowski and Alexander Waibel. 2024. From text segmentation to smart chaptering: A novel benchmark for structuring video transcriptions. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 406–419, St.… find
  28. Fabian Retkowski and Alexander Waibel. 2025. Paragraph segmentation revisited: Towards a standard task for structuring speech. Preprint, arXiv:2512.24517.
  29. Hendrik Schröter, Alberto N. Escalante-B., Tobias Rosenkranz, and Andreas Maier. 2022. DeepFilter-Net: A low complexity speech enhancement framework for full-band audio based on deep filtering. find
  30. In ICASSP 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. on Parsimony and Learning, volume 234 of Proceedings of Machine Learning Research, pages 202–227. PMLR. find
  31. Eduardo R. Soares and Eduardo Barrére. 2018. Automatic topic segmentation for video lectures using low and high-level audio features. In Proceedings of the 24th Brazilian Symposium on Multimedia and the Web, WebMedia ’18, page 189–196, New York, NY, USA. Association for Computing Machinery. find
  32. Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, and Xueqi Cheng. 2019. Outline generation: Understanding the inherent content structure of documents. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR’19, page 745–754, New Yor… find
  33. Lucas Ventura, Antoine Yang, Cordelia Schmid, and Gül Varol. 2025. Chapter-llama: Efficient chaptering in hour-long videos with llms. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, pages 18947–18958. Computer Vision Foundation / IE… find
  34. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In Proceedings of the International Conference on Learning Representations (ICLR). find
  35. Apoorv Vyas, Heng-Jui Chang, Cheng-Fu Yang, Po-Yao Huang, Luya Gao, Julius Richter, Sanyuan Chen, Matt Le, Piotr Dollár, Christoph Feichtenhofer, Ann Lee, and Wei-Ning Hsu. 2025. Pushing the frontier of audiovisual perception with large-scale multimodal correspondence learning. Preprint, arXiv:2512.… — [arXiv:2512.19687](https://arxiv.org/abs/2512.19687) find
  36. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, Neu… find
  37. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025a. Qwen2.5-omni technical report. Preprint, arXiv:2503.20215.
  38. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025b. Qwen3-omni technical report. Preprint, arXiv:2509.17765.
  39. Antoine Yang, Arsha Nagrani, Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023. Vidchapters-7m: Video chapters at scale. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2… find
  40. Ozan Raşit Yürüm, Tuğba Taşkaya-Temizel, and Soner Yıldırım. 2022. The use of video clickstream data to predict university students’ test performance: A comprehensive educational data mining approach. Education and Information Technologies, 28(5):5209–5240. find
  41. Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. 2024. Investigating the catastrophic forgetting in multimodal large language model fine-tuning. In Proceedings of the Conference find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/retkowski-2026-beyond-transcripts-renewed-perspective-audio/