CrossOracle: An Agentic Framework for Real-Time Expert Personas with Parallelized Acoustic Synthesis
Summary
CrossOracle is a deployed cascaded spoken dialogue system for real-time voice conversation in Hinglish (Hindi-English code-switching). It offers eleven expert personas, such as Doctor, Lawyer and Astrologer, plus a user-defined persona option. The author targets the delay that builds up when ASR, an LLM and TTS run in sequence. The paper argues that a cascade is more controllable and cheaper than end-to-end speech-to-speech models like AudioPaLM, which also handle mixed-language input poorly. The system is a Node.js WebSocket layer over Azure Cognitive Services and ElevenLabs Turbo v2.5. It cuts the streaming LLM output into clause-sized pieces using script-aware rules and synthesises them in a background queue while generation continues. An abort mechanism cancels all pending work when the user interrupts. A filter rejects ASR outputs dominated by spurious Latin text. Server logs from 8,000 turns show faster first audio and lower RTF than a fixed-size buffering baseline, with uninterrupted playback. The work offers a practical recipe for low-latency code-switched voice agents, although perceived speech quality (MOS) has not yet been evaluated.
Key points
- The checkFlush() chunking heuristic sends text to TTS on three triggers. In priority order, these are a Devanagari danda, Latin sentence-final punctuation (with an abbreviation blocklist), and newlines, dashes or ellipses. Chunks must be at least 30 characters and are capped at 220.
- Across 8,000 turns, CrossOracle reached a mean TTFB of 4.9 s, compared with 6.5 s for a naive fixed-token buffer. Mean TTS RTF was 0.107 versus 0.134.
- Barge-in handling succeeded in 100% of cases, with server-side abort taking under 1 ms. Cached persona greetings play in about 866 ms.
- ASR is fixed to the hi-IN locale. Transcripts with a Devanagari character ratio below 0.1 and length above 8 characters are discarded. Accepted transcripts averaged a ratio of 0.61, while hallucinated Latin output averaged about 0.02.
- One edge case broke the playback guarantee: a lone emoji token produced an RTF of 2.513. The author has since added a regex filter that strips such characters before synthesis.
Cite
@inproceedings{singh2026crossoracle,
title = {{CrossOracle: An Agentic Framework for Real-Time Expert Personas with Parallelized Acoustic Synthesis}},
author = {Singh, Yash Raj},
booktitle = {Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
year = {2026},
doi = {10.18653/v1/2026.sigdial-1.18},
url = {https://doi.org/10.18653/v1/2026.sigdial-1.18}
}References
- Jaspreet Kaur, Gurpreet Singh, and Rajneesh Sharma. 2020. A review of Devanagari script and its challenges. Journal of Indic Languages. find
- Stephen C Levinson. 2015. Turn-taking in human communication: Origins and implications for language processing. Trends in Cognitive Sciences, 20(1):6– 14. find
- Kentaro Mitsui, Soumi Maiti, and Shinji Watanabe. 2024. PSLM: Parallel generation of text and speech with LLMs for low-latency spoken dialogue systems. arXiv preprint arXiv:2406.12428. find
- OpenAI. 2023. GPT-4 technical report. Technical report, OpenAI. find our card
- Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning. find our card
- Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2021. FastSpeech 2: Fast and high-quality end-to-end text to speech. In find
This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2026-crossoracle-agentic-framework-real-time/