Elamur Research⌕
☰
home / papers / 2020 / 100,000 Podcasts: A Spoken English Document Corpus

100,000 Podcasts: A Spoken English Document Corpus

Paper · 2020 · Ann Clifton, Sravana Reddy, Yongze Yu et al.

Summary

The paper tackles a gap in spoken-language resources. Existing transcribed speech collections are mostly clean read speech, broadcast news, or narrow conversational domains. None captures the mix of interviews, monologue, banter, and non-speech audio typical of podcasts. Spotify researchers released the Spotify Podcast Dataset: just over 100,000 English episodes, close to 60,000 hours of audio, sampled from a little over a year of publishing on the platform. Each episode comes with automatic transcripts from Google Cloud Speech-to-Text (word timings, diarization, punctuation) and creator metadata such as episode descriptions. The authors describe the corpus and benchmark two tasks, passage search and summarization. For summarization they treat creator descriptions as noisy references and compare TextRank, a first-minute baseline, and BART models, scoring outputs with both ROUGE and human judgements. A BART model fine-tuned on podcasts scores best and comes close to human-written descriptions in human ratings. The main contribution is a very large, stylistically varied, noisy spoken corpus that exposes how standard NLP and IR methods behave outside clean written text.

Key points

  • Scale and sources: 105,360 sampled episodes from 18,376 shows, published between January 2019 and March 2020. About 10% come from professional creators and the rest from amateurs. Average episode length is 33.8 minutes and about 5,700 words.
  • Sampling filters:
    • English only, based on creator tags plus langid.py run on descriptions. Some non-English and code-switched material still slipped through.
    • Non-professional episodes over 90 minutes were removed.
    • Shows averaging under 50% speech were dropped.
  • Transcript quality:
    • A sampled word error rate of 18.1% and named-entity accuracy of 81.8%.
    • Diarization is unreliable: in 20 manually checked episodes, 11 had the wrong speaker count and 4 more had boundary errors.
  • Register and summarization ratings:
    • First-person pronouns make up 4.3% of tokens, against 0.40–2.6% across Brown corpus genres. Amplifiers are also far more frequent, marking a conversational style.
    • BART-PODCASTS reached ROUGE-1 F of 28.24 on the "brass" (filtered-description) test subset, versus 20.67 for BART-CNN.
    • In human ratings, BART-PODCASTS was judged Good or better 47% / 55% of the time (brass / non-brass), close to creator descriptions at 50% / 58%. TextRank was rated lowest.

Interesting ideas

  • I1. When references are untrustworthy, the authors split test items by human-rated reference quality and confirm that the system ranking stays the same in each bucket, which gives evidence the automatic metric is still informative.
  • I2. Human assessors scored summaries against the source transcript rather than against a gold summary, so the references themselves (creator descriptions) could be graded alongside system outputs.
  • I3. Instead of discarding noisy found labels (creator descriptions), they applied a cheap quality filter to carve out a cleaner "brass" subset for training and testing, keeping the larger noisy pool for comparison.
  • I4. Small manual spot checks (20 episodes) were used to quantify how reliable an automatic annotation layer such as diarization really is before anyone builds on it.
  • I5. Extractive methods suffered from ASR errors and disfluency in the text they selected, whereas abstractive models smoothed over the noise and produced fluent overviews.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova's evaluation could re-rank parser or reader variants separately on high-confidence and low-confidence silver subsets and accept a metric only if the ordering is stable across both.
  • I2 (answer): Mova's answers to fable or textbook questions could be graded directly against the text world, which would also reveal when the "gold" answer key is itself weak.
  • I3 (learn): Silver trees from witness parsers could be tiered the same way, with a filtered subset for training and the remainder kept as a contrast set that shows whether filtering actually helps.
  • I4 (frames): Before trusting automatic speaker or coreference links in dialogue-heavy texts, Mova could audit a small random sample and record the error rate as a known ceiling on downstream frame accuracy.
  • I5 (back): Generating text from Mova's meaning graph, rather than copying source spans, could naturally clean up disfluent or noisy input when summarizing or compressing speech-like text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States184 works · 69,372 citations🎓University of Massachusetts AmherstAmherst Center · United States106 works · 66,692 citations🎓Dublin City UniversityDublin · Ireland70 works · 9,447 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu, Rezvaneh Rezapour, Hamed Bonab, Maria Eskevich, Gareth Jones, Jussi Karlgren, Ben Carterette and Rosie Jones. 2020. 100,000 Podcasts: A Spoken English Document Corpus. Proceedings of the 28th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.coling-main.519

@inproceedings{clifton2020100,
  title = {{100,000 Podcasts: A Spoken English Document Corpus}},
  author = {Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu, Rezvaneh Rezapour, Hamed Bonab, Maria Eskevich, Gareth Jones, Jussi Karlgren, Ben Carterette and Rosie Jones},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.519},
  url = {https://doi.org/10.18653/v1/2020.coling-main.519}
}

References

41 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James Allan, Jaime Carbonell, George. Doddington, Jonathan Yamron, and Yiming Yang. 1998. Topic Detection and Tracking Pilot Study: Final Report. In Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop, Lansdowne. 007. find
  2. Jana Besser, Katja Hofmann, and Martha A Larson. 2008. An Exploratory Study of User Goals and Strategies in Podcast Search. In Proceedings of Workshop-Woche: Lernen, Wissen & Adaptivität (LWA). find
  3. Jana Besser, Martha Larson, and Katja Hofmann. 2010. Podcast search: User goals and retrieval technologies. Online information review. ↗
  4. Aadyot Bhatnagar, Caiming Xiong, Guangsen Wang, Richard Socher, Weiran Wang, and Yingbo Zhou. 2020. An investigation of phone-based subword units for end-to-end speech recognition. In Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH). ↗
  5. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent Dirichlet Allocation. Journal of machine Learning research, 3. ↗
  6. John W. Du Bois and Robert Engebretson. 2005. Santa Barbara corpus of spoken American English. Linguistic Data Consortium. find
  7. Alexandra Canavan, David Graff, and George Zipperlen. 1997. Callhome American English speech. Linguistic Data Consortium. find
  8. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans. ACL.
  9. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22. ↗
  10. Maria Eskevich, Walid Magdy, and Gareth JF Jones. 2012. New metrics for meaningful evaluation of informally structured speech retrieval. In European Conference on Information Retrieval (ECIR). Springer. find
  11. Marcello Federico and Gareth JF Jones. 2003. find
  12. The CLEF 2003 cross-language spoken document retrieval track. In Workshop of the Cross-Language Evaluation Forum for European Languages, pages 646–652. Springer. find
  13. W Nelson Francis and Henry Kučera. 1967. Computational analysis of present-day American English. Brown University Press, Providence, RI. find
  14. John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, David S Pallett, Nancy L Dahlgren, and Victor Zue. 1990. TIMIT acoustic-phonetic continuous speech corpus. Linguistic Data Consortium. find
  15. John S Garofolo, Cedric GP Auzanne, and Ellen M Voorhees. 2000. The TREC Spoken Document Retrieval Track: A Success Story. NIST SPECIAL PUBLICATION, 500(246). find
  16. John S. Garofolo, Jonathan Fiscus, and Audrey Le. 2004a. Rich Transcription Broadcast News and Conversational Telephone Speech. Linguistic Data Consortium. find
  17. John S Garofolo, Christophe Laprun, Martial Michel, Vincent M Stanford, and Elham Tabassi. 2004b. The NIST meeting room pilot corpus. In Proceedings of the 4th Conference on Language Resources and Evaluation (LREC). European Language Resources Association. find
  18. Demian Gholipour Ghalandari, Chris Hokamp, Nghia The Pham, John Glover, and Georgiana Ifrim. 2020. A large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal. In Proceedings of the 58th Meeting of the Association for Computational Linguistics (ACL). ACL. find
  19. John J. Godfrey and Edward Holliman. 1993. Switchboard-1 release 2. Linguistic Data Consortium. find
  20. Yoichiro Hasebe. 2015. Design and implementation of an online corpus of presentation transcripts of TED talks. Procedia-Social and Behavioral Sciences, 198. find
  21. Charles T Hemphill, John J Godfrey, and George R Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop, Hidden Valley, Pennsylvania. find
  22. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems (NIPS). Curran Associates. ↗
  23. Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, and Chuck Wooters. 2003. The ICSI Meeting Corpus. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP ’03). ↗
  24. Rosie Jones, Ben Carterette, Ann Clifton, Maria Eskevich, Gareth Jones, Jussi Karlgren, Aasish Pappu, Sravana Reddy, and Yongze Yu. 2020. Overview of the TREC 2020 Podcasts Track. In The 29th Text Retrieval Conference (TREC 2020) Notebook. NIST. find
  25. Konstantinos Koumpis and Steve Renals. 2005. Automatic summarization of voicemail messages using lexical and prosodic features. ACM Transactions on Speech and Language Processing (TSLP), 2. ↗
  26. Victor Lavrenko and W. Bruce Croft. 2017. Relevance-based language models. SIGIR Forum, 51(2). ↗
  27. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). ACL.
  28. Feifan Liu and Yang Liu. 2008. Correlation between ROUGE and Human Evaluation of Extractive Meeting Summaries. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies (ACL), Columbus, Ohio. ACL. find
  29. Iain Mccowan, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Wilfried Post, Dennis Reidsma, and Pierre Wellner. 2005. The AMI meeting corpus. In Proceedings of Measuring Behavior 2005, 5th International Conference on Methods and Techniques in Behavioral Research. Noldus. find
  30. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on Empirical Methods in Natural Language Processing (EMNLP). ACL. find
  31. Derek Miller. 2019. arXiv:1906.04165. Leveraging BERT for Extractive Text Summarization on Lectures. arXiv preprint Randolph Quirk, Sidney Greenbaum, Geoffrey Leech, and Jan Svartvik. 1985. A comprehensive grammar of contemporary English. London: Longman. ↗
  32. Damiano Spina, Johanne R. Trippas, Lawrence Cavedon, and Mark Sanderson. 2017. Extracting audio summaries to support effective spoken document search. Journal of the Association for Information Science and Technology, 68(9). ↗
  33. Josef Steinberger and Karel Jezek. 2004. Using Latent Semantic Analysis in Text Summarization and Summary Evaluation. In Proceedings of International Conference on Information Systems Implementation and Modelling. find
  34. Paul Tardy, David Janiszek, Yannick Estève, and Vincent Nguyen. 2020. Align then Summarize: Automatic Alignment Methods for Summarization Corpus Creation. In Proceedings of The 12th Language Resources and Evaluation Conference (LREC). European Language Resources Association. ↗
  35. Egidio Terra and Robert Warren. 2005. Poison pills: harmful relevant documents in feedback. In Proceedings of the 14th ACM international conference on Information and knowledge management CIKM. ACM. find
  36. Ellen M Voorhees and Donna K Harman. 2005. TREC: Experiment and evaluation in information retrieval, volume 63. MIT press, Cambridge. find
  37. Ellen M Voorhees. 2000. Variations in relevance judgments and the measurement of retrieval effectiveness. Information processing & management, 36. ↗
  38. Gavin Whitner. 2020. The meteoric rise of podcasting. https://musicoomph.com/podcast-statistics (Accessed July 2020). ↗
  39. Wen Xiao and Giuseppe Carenini. 2019. Extractive Summarization of Long Documents by Combining Global and Local Context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong. ACL. ↗
  40. Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1253–1256. find
  41. Chujie Zheng, Harry Jiannan Wang, Kunpeng Zhang, and Ling Fan. 2020. A Baseline Analysis for Podcast Abstractive Summarization. arXiv preprint arXiv:2008.10648. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/clifton-2020-100-000-podcasts-spoken-english/