Elamur Research⌕
☰
home / papers / 2026 / J-CHAT: Japanese Large-scale Spoken Dialogue Corpus for Spoken Dialogue Language Modeling

J-CHAT: Japanese Large-scale Spoken Dialogue Corpus for Spoken Dialogue Language Modeling

Summary

End-to-end spoken dialogue systems (SDSs) model speech directly instead of chaining ASR, text generation and TTS. They need tens of thousands of hours of conversational audio that is clean, varied and spontaneous. Few such resources are openly available. Large speech corpora built for ASR or TTS are cut into isolated utterances, which loses turn-taking and coherence across turns. Studio-recorded alternatives such as Seamless Interaction are expensive to reproduce in new languages.

The authors build a fully automatic, language-independent pipeline:

  • collect audio from YouTube and Japanese podcasts,
  • keep Japanese speech using language identification,
  • group speech into multi-speaker conversations with speaker diarization,
  • remove background music with a source-separation model,
  • transcribe the result with ASR.

Applied to Japanese, the pipeline produces J-CHAT, an openly released dialogue-level corpus of about 76k hours. The authors trained generative spoken dialogue language models on it and report that the corpus is useful for SDS development. J-CHAT gives Japanese a dialogue corpus comparable in scale to major speech datasets, and the method can be reused for other languages.

Key points

  • Size and sources: J-CHAT contains 76,036 hours and 5,424,514 dialogues. The podcast subset has 65,019 h and the YouTube subset 11,017 h. A dialogue averages about 50 s, 10.1 turns and 3.14 speakers. The corpus is over 1,000 times larger than CallHome-JP (49 h).
  • Raw collection: About 600k YouTube files (~180k h) were found by searching random Wikipedia titles. About 880k podcast files (~140k h) came from RSS feeds labeled Japanese in PodcastIndex.
  • Language filtering: Whisper language ID kept segments with a Japanese probability above 0.8. This retained 55.7% of the YouTube audio and 84.7% of the podcast audio.
  • Dialogue extraction: PyAnnote diarization was used. Audio was split into dialogues at pauses of 5 s or more. Dialogues where one speaker held over 80% of the time were discarded as monologues.
  • Cleaning, transcripts and license: Demucs removed background music. reazonspeech-nemo-v2 produced transcripts with subword alignments. The corpus is published on Hugging Face (sarulab-speech/J-CHAT) under CC BY-NC 4.0 and is intended for information analysis, as allowed by Japan's copyright law (Article 30-4).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2rjcajopifoh

@inproceedings{nakata2026j,
  title = {{J-CHAT: Japanese Large-scale Spoken Dialogue Corpus for Spoken Dialogue Language Modeling}},
  author = {Nakata, Wataru and Seki, Kentaro and Yanaka, Hitomi and Saito, Yuki and Takamichi, Shinnosuke and Saruwatari, Hiroshi},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2rjcajopifoh},
  url = {https://doi.org/10.63317/2rjcajopifoh}
}

References

33 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vasu Agrawal, Akinniyi Akinyemi, Kathryn Alvero, Morteza Behrooz, Julia Buffalini, Fabio Maria Carlucci, Joy Chen, Junming Chen, Zhang Chen, Shiyang Cheng, et al. 2025. Seamless interaction: Dyadic audiovisual motion modeling and large-scale dataset. arXiv:2506.22554. arXiv preprint Canavan Alexandr… — [arXiv:2506.22554](https://arxiv.org/abs/2506.22554) find
  2. Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. 2023. Soundstorm: Efficient parallel audio generation. find
  3. Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Zhao You, and Zhiyong Yan. 2021. GigaSpeech: An Evolving, Mu… find
  4. Interspeech 2021, pages 3670– 3674. find
  5. Christopher Cieri, David Miller, and Kevin Walker. 2004. The Fisher Corpus: a resource for the next generations of speech-to-text. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal. European Language Resources Association (ELRA). find our card
  6. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. arXiv, arXiv:2410.00037. find our card
  7. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460. find our card
  8. Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. 2024. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38… find
  9. Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. 2022. Text-free prosody-aware generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for… find
  10. Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Michiel Bacchiani, Yu Zhang, Wei Han, and Ankur Bapna. 2023. LibriTTS-R: A restored multi-speaker text-to-speech corpus. In Proc. find
  11. INTERSPEECH 2023, pages 5496–5500. find
  12. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. In Advances in Neural Information Processing Systems, volume 33, pages 17022–17033. Curran Associates, Inc. find
  13. Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. 2021. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336–1354. find
  14. Keon Lee, Kyumin Park, and Daeyoung Kim. 2023. DailyTalk: Spoken dialogue dataset for conversational text-to-speech. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. find
  15. Kentaro Mitsui, Yukiya Hono, and Kei Sawada. 2023. Towards human-like spoken dialogue generation between ai agents from written dialogue. find
  16. Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian Möller. 2021. NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets. In Proc. find
  17. Interspeech 2021, pages 2127–2131. find
  18. Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdelrahman Mohamed, and Emmanuel Dupoux. 2023. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11:250–266. find
  19. Se Park, Chae Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeonghun Yeo, and Yong Ro. 2024. Let’s go real talk: Spoken dialogue model for face-to-face conversation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16334–16348.… find
  20. Alexis Plaquet and Hervé Bredin. 2023. Powerset multi-class cross entropy loss for neural speaker diarization. In Proc. find
  21. INTERSPEECH 2023, pages 3222–3226. find
  22. Simon Rouard, Francisco Massa, and Alexandre Défossez. 2023. Hybrid transformers for music source separation. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. find
  23. Yuki Saito, Yuto Nishimura, Shinnosuke Takamichi, Kentaro Tachibana, and Hiroshi Saruwatari. 2022. STUDIES: Corpus of Japanese Empathetic Dialogue Speech Towards Friendly Voice Agent. In Proc. find
  24. Interspeech 2022, pages 5155– 5159. find
  25. Kei Sawada, Tianyu Zhao, Makoto Shing, Kentaro Mitsui, Akio Kaga, Yukiya Hono, Toshiaki Wakatsuki, and Koh Mitsuda. 2024. Release of pre-trained models for the Japanese language. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation… find
  26. Kentaro Seki, Shinnosuke Takamichi, Takaaki Saeki, and Hiroshi Saruwatari. 2023. Text-to-speech synthesis from dark data with evaluation-in-the-loop data selection. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1–5. find
  27. Zaid Sheikh, Shuichiro Shimizu, Siddhant Arora, Jiatong Shi, Samuele Cornell, Xinjian Li, and Shinji Watanabe. 2025. Scalable spontaneous speech dataset (SSSD): Crowdsourcing data collection to promote dialogue research. In Proc. find
  28. Interspeech 2025, pages 3963–3967. find
  29. David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. 2018. X-Vectors: Robust dnn embeddings for speaker recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5329–5333. find
  30. Shinnosuke Takamichi, Ludwig Kürzinger, Takaaki Saeki, Sayaka Shiota, and Shinji Watanabe. 2021. JTubeSpeech: corpus of japanese speech collected from youtube for speech recognition and speaker verification. find
  31. Shinnosuke Takamichi, Ryosuke Sonobe, Kentaro Mitsui, Yuki Saito, Tomoki Koriyama, Naoko Tanji, and Hiroshi Saruwatari. 2020. JSUT and JVS: Free japanese voice corpora for accelerating speech synthesis research. Acoustical Science and Technology, 41(5):761–768. find
  32. Detai Xin, Junfeng Jiang, Shinnosuke Takamichi, Yuki Saito, Akiko Aizawa, and Hiroshi Saruwatari. 2024. JVNV: A corpus of japanese emotional speech with verbal content and nonverbal expressions. IEEE Access, 12:19752–19764. find
  33. Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019. LibriTTS: A corpus derived from LibriSpeech for text-to-speech. arXiv preprint arXiv:1904.02882. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/nakata-2026-j-chat-japanese-large-scale/