Elamur Research⌕
☰
home / papers / 2026 / F-Actor: Controllable Conversational Behavior in Full-Duplex Models

F-Actor: Controllable Conversational Behavior in Full-Duplex Models

Paper · 2026 · Maike Züfle, Ondrej Klejch, Nicholas Sanders et al.

Summary

Full-duplex spoken dialogue systems can listen and talk simultaneously, but they offer users little control over how they behave. They are usually evaluated on how they react to user overlaps rather than on whether they backchannel or interrupt themselves, and comparable instruction-following systems (MinMo, Voila, BeDLM) have not been publicly released. The authors present F-Actor, which they describe as the first open, controllable full-duplex speech model. It takes a prompt and a speaker embedding that set the system's voice, the conversation topic or narrative, target counts of backchannels and interruptions, and whether the system opens the dialogue. The design is built for academic budgets: Llama3.2-1B-Instruct is fine-tuned together with audio heads in a single training stage, while the NanoCodec encoder (FSQ, four independent codebooks) stays frozen. Training uses about 2,000 hours of synthetic (TTS) English dialogue. The model produces coherent conversations whose turn-taking statistics resemble the training data and baselines such as Moshi. Requested backchannel and interruption counts act only as a directional signal. Model and code are released for reproducible research.

Key points

  • Architecture: Llama3.2-1B-Instruct is the backbone, and Nemo-nano-codec-22khz-0.6kbps-12.5fps produces 4 FSQ codebooks of size 4032 at 12.5 fps. Because the codebooks are independent, they can be predicted in parallel without a depth transformer.
  • Compute: Training takes two days on four A100-40GB GPUs, with no large-scale pretraining or multi-stage optimization.
  • Best configuration and turn-taking: The best setup uses word-level text alignment, an audio delay of 2, and sampling temperature 0.9 (the best quality–dynamics trade-off). Per minute of dialogue it yields about 59.3s of IPUs, 10.4s of pauses, 3.0s of gaps and 5.4s of overlap. Pauses exceed gaps, as in natural speech.
  • Codec ablation: Replacing FSQ with an RVQ codec gives comparable dialogue behaviour, but speech quality drops sharply (UTMOS 2.5).
  • Limitations:
    • The model produces fewer backchannels and interruptions than requested.
    • NanoCodec lacks streaming encoding, which forces chunk-wise processing.
    • The model covers English only.
    • Voices are limited to a fixed embedding pool, partly as a safeguard against impersonation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom381 works · 80,747 citations🎓Karlsruhe Institute of TechnologyKarlsruhe · Germany13 works · 834 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 654 works

Cite

Cite as Maike Züfle, Ondrej Klejch, Nicholas Sanders, Jan Niehues, Alexandra Birch and Tsz Kin Lam. 2026. F-Actor: Controllable Conversational Behavior in Full-Duplex Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.242

@inproceedings{zufle2026f,
  title = {{F-Actor: Controllable Conversational Behavior in Full-Duplex Models}},
  author = {Maike Züfle, Ondrej Klejch, Nicholas Sanders, Jan Niehues, Alexandra Birch and Tsz Kin Lam},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.242},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.242}
}

References

59 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, and 1 others. 2024. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430.
  2. Siddhant Arora, Kai-Wei Chang, Chung-Ming Chien, Yifan Peng, Haibin Wu, Yossi Adi, Emmanuel Dupoux, Hung-Yi Lee, Karen Livescu, and Shinji Watanabe. 2025a. On the landscape of spoken language models: A comprehensive survey. Preprint, arXiv:2504.08528. — [arXiv:2504.08528](https://arxiv.org/abs/2504.08528) find
  3. Edresson Casanova, Paarth Neekhara, Ryan Langman, Shehzeen Hussain, Subhankar Ghosh, Xuesong Yang, Ante Jukic, Jason Li, and Boris Ginsburg. 2025. Nanocodec: Towards high-quality ultra fast speech LLM inference. In 26th Annual Conference of the International Speech Communication Association, Intersp… find
  4. Junjie Chen, Yao Hu, Junjie Li, Kangyue Li, Kun Liu, Wenpeng Li, Xu Li, Ziyuan Li, Feiyu Shen, Xu Tang, Manzhen Wei, Yichen Wu, Fenglong Xie, Kaituo Xu, and Kun Xie. 2025a. Fireredchat: A pluggable, full-duplex voice interaction system with cascaded and semi-cascaded implementations. Preprint, arXiv… — [arXiv:2509.06502](https://arxiv.org/abs/2509.06502) find
  5. Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, Yabin Li, Xiang Lv, Jiaqing Liu, Haoneng Luo, Bin Ma, Chongjia Ni, Xian Shi, Jialong Tang, Hui Wang, and 17 others. 2025b. Minmo: A multimodal large language model for seamless… — [arXiv:2501.06282](https://arxiv.org/abs/2501.06282) find
  6. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, and 1 others. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518. find
  7. Yuxuan Chen and Haoyuan Yu. 2025. From turn-taking to synchronous dialogue: A survey of full-duplex spoken language models. Preprint, arXiv:2509.14515. — [arXiv:2509.14515](https://arxiv.org/abs/2509.14515) find
  8. Myra Cheng, Kristina Gligoric, Tiziano Piccardi, and Dan Jurafsky. 2024. AnthroScore: A computational linguistic measure of anthropomorphism. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 807–825, St. Jul… find
  9. Eugene Cho, Nasim Motalebi, S. Shyam Sundar, and Saeed Abdullah. 2022. Alexa as an active listener: How backchanneling can elicit self-disclosure and promote user experience. Proc. ACM Hum.-Comput. Interact., 6(CSCW2). find
  10. Wenqian Cui, Lei Zhu, Xiaohui Li, Zhihan Guo, Haoli Bai, Lu Hou, and Irwin King. 2025. Think before you talk: Enhancing meaningful dialogue generation in full-duplex speech language models with planning-inspired text guidance. Preprint, arXiv:2508.07375. — [arXiv:2508.07375](https://arxiv.org/abs/2508.07375) find
  11. Siddhant Arora, Jinchuan Tian, Hayato Futami, Jiatong Shi, Yosuke Kashiwagi, Emiru Tsunoo, and Shinji Watanabe. 2025b. Chain-of-thought reasoning in streaming full-duplex end-to-end spoken dialogue systems. Preprint, arXiv:2510.02066. — [arXiv:2510.02066](https://arxiv.org/abs/2510.02066) find
  12. Nauman Dawalatabad, Mirco Ravanelli, François Grondin, Jenthe Thienpondt, Brecht Desplanques, and Hwidong Na. 2021. Ecapa-tdnn embeddings for speaker diarization. In Proc. find
  13. Interspeech 2021, pages 3560–3564. find
  14. Hervé Bredin. 2023. pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. In Proc. find
  15. Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022. High fidelity neural audio compression. Transactions on Machine Learning Research.
  16. Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijie Yan. 2024. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. Preprint, arXiv:2407.05407.
  17. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. Preprint, arXiv:2410.00037.
  18. Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, Long Ma, Xiawu Zheng, Rongrong Ji, Xing Sun, Caifeng Shan, and Ran He. 2025. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. Preprint, arXiv:2501.01957. — [arXiv:2501.01957](https://arxiv.org/abs/2501.01957) find
  19. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  20. Mattias Heldner and Jens Edlund. 2010. Pauses, gaps and overlaps in conversations. Journal of Phonetics, 38(4):555–568. find
  21. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. CoRR, abs/2106.07447.
  22. Ke Hu, Ehsan Hosseini-Asl, Chen Chen, Edresson Casanova, Subhankar Ghosh, Piotr Żelasko, Zhehuai Chen, Jason Li, Jagadeesh Balam, and Boris Ginsburg. 2025. Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model. In Interspeech 2025, pages 2715–2719. find
  23. Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, and 1 others. 2022. Text-free prosody-aware generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Com…
  24. Katharina Kühne, Martin H. Fischer, and Yuefang Zhou. 2021. The human takes it all. find
  25. Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9…
  26. Sehun Lee, Kang-wook Kim, and Gunhee Kim. 2025. Behavior-SD: Behaviorally aware spoken dialogue generation with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1:… find
  27. Yaniv Leviathan and Yossi Matias. 2018. Google duplex: An ai system for accomplishing real-world tasks over the phone. find
  28. Borui Liao, Yulong Xu, Jiao Ou, Kaiyuan Yang, Weihua Jian, Pengfei Wan, and Di Zhang. 2025. Flex-duo: A pluggable system for enabling full-duplex capabilities in speech dialogue systems. Preprint, arXiv:2502.13472. — [arXiv:2502.13472](https://arxiv.org/abs/2502.13472) find
  29. Guan-Ting Lin, Shih-Yun Shan Kuan, Qirui Wang, Jiachen Lian, Tingle Li, Shinji Watanabe, and Hung yi Lee. 2025. Full-duplex-bench v1.5: Evaluating overlap handling for full-duplex speech models. Preprint, arXiv:2507.23159. — [arXiv:2507.23159](https://arxiv.org/abs/2507.23159) find
  30. Ting-En Lin, Yuchuan Wu, Fei Huang, Luo Si, Jian Sun, and Yongbin Li. 2022. Duplex conversation: Towards human-like interaction in spoken dialogue systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, page 3299–3308. ACM. find
  31. John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude E. Shannon. 2006. A proposal for the dartmouth summer research project on artificial intelligence, august 31, 1955. AI Mag., 27(4):12–14. find
  32. Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. 2024. Finite scalar quantization: Vq-vae made simple. In International Conference on Representation Learning, volume 2024, pages 51772–51783. find
  33. Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdelrahman Mohamed, and Emmanuel Dupoux. 2023. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11:250–266.
  34. Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, and Emmanuel Dupoux. 2025. SpiRit-LM: Interleaved spoken and…
  35. Atsumoto Ohashi, Shinya Iizuka, Jingjing Jiang, and Ryuichiro Higashinaka. 2025. Towards a Japanese Full-duplex Spoken Dialogue System. In Interspeech 2025, pages 1783–1787. find
  36. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an ASR corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. find
  37. Yizhou Peng, Yi-Wen Chao, Dianwen Ng, Yukun Ma, Chongjia Ni, Bin Ma, and Eng Siong Chng. 2025. FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systems. In Interspeech 2025, pages 176–180. find
  38. Alexis Plaquet and Hervé Bredin. 2023. Powerset multi-class cross entropy loss for neural speaker diarization. In Proc. find
  39. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, and 1 others. 2011. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding, volume 1, pages 5–… find
  40. Antoine Raux and Maxine Eskenazi. 2012. Optimizing the turn-taking behavior of task-oriented spoken dialog systems. ACM Trans. Speech Lang. Process., 9(1). find
  41. Robin Ruede, Markus Müller, Sebastian Stüker, and Alex Waibel. 2017. Yeah, right, uh-huh: A deep learning backchannel predictor. Preprint, arXiv:1706.01340. — [arXiv:1706.01340](https://arxiv.org/abs/1706.01340) find
  42. Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022. In Interspeech 2022, pages 4521–4525.
  43. Nicholas Sanders, Yuanchao Li, Korin Richmond, and Simon King. 2025. Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information. In Interspeech 2025, pages 5403–5407. find
  44. Emanuel Schegloff. 1982. Discourse as an interactional achievement: Some uses of ‘uh huh’ and other things that come between sentences, pages 71–93. Georgetown University Press. find
  45. Yemin Shi, Yu Shu, Siwei Dong, Guangyi Liu, Jaward Sesay, Jingwen Li, and Zhiting Hu. 2025. Voila: Voice-language foundation models for real-time autonomous interaction and voice role-play. Preprint, arXiv:2505.02707. — [arXiv:2505.02707](https://arxiv.org/abs/2505.02707) find
  46. Tanya Stivers, Nick J. Enfield, Penelope Brown, Christina Englert, Makoto Hayashi, Trine Heinemann, Gertie Hoymann, Federico Rossano, Jan Peter De Ruiter, Kyung-Eun Yoon, Stephen C. Levinson, Paul Kay, and Krishna Y. 2009. Universals and cultural variation in turn-taking in conversation. Proceedings… find
  47. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Sl… find
  48. Silero Team. 2024. Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier. https://github.com/ snakers4/silero-vad. ↗
  49. Bandhav Veluri, Benjamin N Peloquin, Bokai Yu, Hongyu Gong, and Shyamnath Gollakota. 2024. Beyond turn-based interfaces: Synchronous LLMs as full-duplex dialogue agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21390–21402, Miami, Florida, USA.… find
  50. Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, and 1 others. 2023. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111.
  51. Peng Wang, Songshuo Lu, Yaohua Tang, Sijie Yan, Wei Xia, and Yuanjun Xiong. 2024a. A full-duplex speech dialogue scheme based on large language model. In Advances in Neural Information Processing Systems, volume 37, pages 13372–13403. Curran Associates, Inc. find
  52. Qichao Wang, Ziqiao Meng, Wenqian Cui, Yifei Zhang, Pengcheng Wu, Bingzhe Wu, Irwin King, Liang Chen, and Peilin Zhao. 2025. Ntpp: Generative speech language modeling for dual-channel spoken dialogue via next-token-pair prediction. Preprint, arXiv:2506.00975. — [arXiv:2506.00975](https://arxiv.org/abs/2506.00975) find
  53. Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. 2024b. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm. Preprint, arXiv:2411.00774. find
  54. Dan Wells, Hao Tang, and Korin Richmond. 2022. Phonetic analysis of self-supervised representations of english speech. In Interspeech 2022, pages 3583– 3587. find
  55. Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, and Chao Zhang. 2024. Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation. Preprint, arXiv:2411.18138. — [arXiv:2411.18138](https://arxiv.org/abs/2411.18138) find
  56. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2021. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507.
  57. Hao Zhang, Weiwei Li, Rilin Chen, Vinay Kothapally, Meng Yu, and Dong Yu. 2025a. Llm-enhanced dialogue management for full-duplex spoken dialogue systems. Preprint, arXiv:2502.14145. — [arXiv:2502.14145](https://arxiv.org/abs/2502.14145) find
  58. Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, Chao-Hong Tan, Zhihao Du, and ShiLiang Zhang. 2025b. OmniFlatten: An end-to-end GPT model for seamless voice conversation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Ling…
  59. Vilém Zouhar and Tom Kocmi. 2026. Pearmut: Human evaluation of translation made trivial. Preprint, arXiv:2601.02933. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zufle-2026-f-actor-controllable-conversational-behavior/