Elamur Research⌕
☰
home / papers / 2026 / DialogueSidon: Recovering Full-Duplex Dialogue Tracks from In-the-Wild Dialogue Audio

DialogueSidon: Recovering Full-Duplex Dialogue Tracks from In-the-Wild Dialogue Audio

Summary

Spoken dialogue systems that model overlap, backchannels and turn-taking need full-duplex recordings, where each speaker has a separate track. Such data are scarce. Telephone corpora like Fisher are small and costly to collect. Internet conversations are plentiful, but they come as noisy single-channel mixtures. The authors present DialogueSidon, which handles restoration and speaker separation in a single step. It extends the restoration model Sidon. First, a VAE compresses w2v-BERT 2.0 features into a compact latent space. Then a diffusion-based predictor, conditioned on the degraded mixture, estimates one clean latent per speaker, and the VAE decoder converts these back to waveforms. The model was compared against Sidon and against GENESES, both the original checkpoint and a version retrained on conversations. Tests covered English data, the five-language CallFriend corpus, and the in-the-wild OpenDialog corpus. DialogueSidon gave much lower WER and higher human-rated quality. It ran about 60 times faster than GENESES, although automatic quality predictors scored it somewhat lower. This makes it a practical route to large full-duplex corpora built from web audio. Code and a demo are public.

Key points

  • Architecture: a two-stage pipeline. Stage 1 trains an SSL-VAE on layer-8 features of w2v-BERT 2.0. Stage 2 trains a diffusion latent predictor that uses LoRA on the SSL model and permutation matching. A 32-dimensional latent worked best, while 128 dimensions hurt every metric.
  • English results: WER was 14.39%, compared with 33.54% for retrained GENESES and 57.47% for Sidon. The original monologue-trained GENESES reached 79.99%, which shows that training on conversational data is crucial.
  • OpenDialog (in-the-wild):
    • WER was 13.86%, versus 43.79% for retrained GENESES and 74.51% for the original.
    • MOS was 3.708, versus 3.131 and 2.611, with all differences significant.
    • NISQA and DNSMOS were lower than for GENESES.
  • CallFriend (multilingual): covers German, French, Japanese, Spanish and Mandarin. DialogueSidon had the lowest p-CER among separation methods in every language. For example, German was 14.7% versus 52.3% for retrained GENESES.
  • Efficiency: on an H100, the real-time factor (RTF) was 0.010 versus 0.604, a 60.4× speedup. The diffusion model has 88M parameters, compared with 393M for GENESES.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Dialogue systems · Speech & spoken language

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.sigdial-1.1

@inproceedings{nakata2026dialoguesidon,
  title = {{DialogueSidon: Recovering Full-Duplex Dialogue Tracks from In-the-Wild Dialogue Audio}},
  author = {Nakata, Wataru and Saito, Yuki and Yamauchi, Kazuki and Tsunoo, Emiru and Saruwatari, Hiroshi},
  booktitle = {Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2026},
  doi = {10.18653/v1/2026.sigdial-1.1},
  url = {https://doi.org/10.18653/v1/2026.sigdial-1.1}
}

References

51 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. Preprint, arXiv:2410.00037. find our card
  2. Vasu Agrawal, Akinniyi Akinyemi, Kathryn Alvero, Morteza Behrooz, Julia Buffalini, Fabio Maria Carlucci, Joy Chen, Junming Chen, Zhang Chen, Shiyang Cheng, Praveen Chowdary, Joe Chuang, Antony D’Avirro, Jon Daly, Ning Dong, Mark Duppenthaler, Cynthia Gao, Jeff Girard, Martin Gleize, and 65 others. 2… — [arXiv:2506.22554](https://arxiv.org/abs/2506.22554) find
  3. Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. 2022. FSD50K: An open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829–852. find
  4. Jont B. Allen and David A. Berkley. 1979. Image method for efficiently simulating small‐room acoustics. The Journal of the Acoustical Society of America, 65(4):943–950. find
  5. Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pag… find
  6. Kohei Asai, Wataru Nakata, Yuki Saito, and Hiroshi Saruwatari. 2026. Geneses: Unified generative speech enhancement and separation. Preprint, arXiv:2601.18456. — [arXiv:2601.18456](https://arxiv.org/abs/2601.18456) find
  7. John J. Godfrey and Edward Holliman. 1993. Switchboard-1 Release 2. find
  8. Alexandra Canavan and George Zipperlen. 1996a. CALLFRIEND Canadian French. find
  9. Heitor R. Guimarães, Jiaqi Su, Rithesh Kumar, Tiago H. Falk, and Zeyu Jin. 2025. DiTSE: High-fidelity generative speech enhancement via latent diffusion transformers. Preprint, arXiv:2504.09381. — [arXiv:2504.09381](https://arxiv.org/abs/2504.09381) find
  10. Alexandra Canavan and George Zipperlen. 1996b. CALLFRIEND German. find
  11. Alexandra Canavan and George Zipperlen. 1996c. CALLFRIEND Japanese. find
  12. Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu. 2024. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In 2024 IEEE Spoke… find
  13. Alexandra Canavan and George Zipperlen. 1996d. CALLFRIEND Spanish Non-Caribbean Dialect. find
  14. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022. WavLM: Large-scale self-supervised pre-training for full stack… find
  15. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. find
  16. Christopher Cieri, David Miller, and Kevin Walker. 2004. The fisher corpus: a resource for the next generations of speech-to-text. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal. European Language Resources Association (ELRA). find our card
  17. Shigeki Karita, Yuma Koizumi, Heiga Zen, Haruko Ishikawa, Robin Scheibler, and Michiel Bacchiani. 2025. Miipher-2: A universal speech restoration model for million-hour scale data restoration. In 2025 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 1–5. find
  18. Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Michiel Bacchiani, Yu Zhang, Wei Han, and Ankur Bapna. 2023a. find
  19. LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus. In Interspeech 2023, pages 5496–5500. find
  20. Daniele Mirabilii, Alexander Lodermeyer, Felix Czwie-long, Stefan Becker, and Emanuël A.P. Habets. 2022. Simulating wind noise with airflow speed-dependent characteristics. In 2022 International Workshop on Acoustic Signal Enhancement (IWAENC), pages 1–5. find
  21. Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian Möller. 2021. NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets. In Interspeech 2021, pages 2127–2131. find
  22. Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Yu Zhang, Wei Han, Ankur Bapna, and Michiel Bacchiani. 2023b. Miipher: A robust speech restoration model integrating self-supervised speech and text representations. In 2023 IEEE Workshop on Applications of Signal P… find
  23. Wataru Nakata, Yuki Saito, Yota Ueda, and Hiroshi Saruwatari. 2026a. Sidon: Fast and robust open-source multilingual speech restoration for large-scale dataset cleansing. Preprint, arXiv:2509.17052. find
  24. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. In Advances in Neural Information Processing Systems, volume 33, pages 17022–17033. Curran Associates, Inc. find
  25. Wataru Nakata, Kentaro Seki, Hitomi Yanaka, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2026b. J-CHAT: Japanese large-scale spoken dialogue corpus for spoken dialogue language modeling. Preprint, arXiv:2407.15828. find our card
  26. Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. 2023. High-fidelity audio compression with improved RVQGAN. In Thirty-seventh Conference on Neural Information Processing Systems. find
  27. Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdelrahman Mohamed, and Emmanuel Dupoux. 2023. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11:250–266. find
  28. LDC. 2008a. CABank Chinese CallHome Corpus. find
  29. LDC. 2008b. CABank English CallHome Corpus. find
  30. William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4172–4182. find our card
  31. LDC. 2008c. CABank Japanese CallHome Corpus. find
  32. LDC. 2008d. CABank Spanish CallHome Corpus. find
  33. Keon Lee, Kyumin Park, and Daeyoung Kim. 2023. Dailytalk: Spoken dialogue dataset for conversational text-to-speech. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. find
  34. Yifan Peng, Muhammad Shakeel, Yui Sudo, William Chen, Jinchuan Tian, Chyi-Jiunn Lin, and Shinji Watanabe. 2025. OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaning. In Interspeech 2025, pages 2225–2229. find
  35. Haohe Liu, Xubo Liu, Qiuqiang Kong, Qiao Tian, Yan Zhao, DeLiang Wang, Chuanzeng Huang, and Yuxuan Wang. 2022. VoiceFixer: A Unified Framework for High-Fidelity Speech Restoration. In Interspeech 2022, pages 4232–4236. find
  36. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. Preprint, arXiv:2212.04356. find our card
  37. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  38. Chandan K A Reddy, Vishak Gopal, and Ross Cutler. 2021. Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6493–6497. find
  39. Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. 2025. DPM-Solver++: Fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Research, 22(4):730–751. find
  40. Rajarshi Roy, Jonathan Raiman, Sang gil Lee, Teodor-Dumitru Ene, Robert Kirby, Sungwon Kim, Jaehyeon Kim, and Bryan Catanzaro. 2026. Personaplex: Voice and role control for full duplex conversational speech models. Preprint, arXiv:2602.06053. — [arXiv:2602.06053](https://arxiv.org/abs/2602.06053) find
  41. Yi Luo and Nima Mesgarani. 2019. Conv-TasNet: Surpassing ideal time–frequency magnitude masking for speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(8):1256–1266. find
  42. Kohei Saijo, Wangyou Zhang, Samuele Cornell, Robin Scheibler, Chenda Li, Zhaoheng Ni, Anurag Kumar, Marvin Sach, Yihui Fu, Wei Wang, Tim Fingscheidt, and Shinji Watanabe. 2025. find
  43. Interspeech 2025 UR-GENT Speech Enhancement Challenge. In Interspeech 2025, pages 858–862. find
  44. Min Ma, Yuma Koizumi, Shigeki Karita, Heiga Zen, Jason Riesa, Haruko Ishikawa, and Michiel Bacchiani. 2024. FLEURS-R: A Restored Multilingual Speech Corpus for Generation Tasks. In Interspeech 2024, pages 1835–1839. find
  45. Tim Salimans and Jonathan Ho. 2022. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations. find
  46. Zizheng Zhang, Chen Chen, Hsin-Hung Chen, Xiang Liu, Yuchen Hu, and Eng Siong Chng. 2024. Noise-aware speech separation with contrastive learning. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1381–1385. find
  47. Robin Scheibler, Eric Bezzam, and Ivan Dokmanić. 2018. Pyroomacoustics: A python package for audio room simulation and array processing algorithms. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 351–355. find
  48. Han Zhu, Wei Kang, Liyong Guo, Zengwei Yao, Fangjun Kuang, Weiji Zhuang, Zhaoqing Li, Zhifeng Han, Dong Zhang, Xin Zhang, Xingchen Song, Long Lin, and Daniel Povey. 2025. Zipvoice-dialog: Non-autoregressive spoken dialogue generation with flow matching. Preprint, arXiv:2507.09318. find
  49. Robin Scheibler, John R. Hershey, Arnaud Doucet, and Henry Li. 2025. Source separation by flow matching. In 2025 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 1–5. find
  50. Liu Ziyin, Tilman Hartwig, and Masahito Ueda. 2020. Neural networks fail to learn periodic functions and how to fix it. In Advances in Neural Information Processing Systems, volume 33, pages 1583–1594. Curran Associates, Inc. find
  51. Bowen Shi, Andros Tjandra, John Hoffman, Helin Wang, Yi-Chiao Wu, Luya Gao, Julius Richter, Matt Le, Apoorv Vyas, Sanyuan Chen, Christoph Feichtenhofer, Piotr Dollár, Wei-Ning Hsu, and Ann Lee. 2025. SAM Audio: Segment anything in audio. Preprint, arXiv:2512.18099. — [arXiv:2512.18099](https://arxiv.org/abs/2512.18099) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nakata-2026-dialoguesidon-recovering-full-duplex-dialogue/