Elamur Research⌕
☰
home / papers / 2026 / PlanRAG-Audio: Planning and Retrieval Augmented Generation for Long-form Audio Understanding

PlanRAG-Audio: Planning and Retrieval Augmented Generation for Long-form Audio Understanding

Paper · 2026 · Masao Someki, Chien-yu Huang, Siddhant Arora et al.

Summary

Large audio language models (LALMs) struggle with recordings that last minutes or hours. The token count becomes very large, and many questions require combining different signals spread across the recording: what was said, who said it, emotional tone, and non-speech sounds. The authors present PlanRAG-Audio, which recasts long-form audio understanding as structured information retrieval. The pipeline has four stages:

  1. Pretrained perception modules (ASR, speaker diarization, emotion recognition, sound event detection) turn the audio into separate time-stamped streams stored in a database.
  2. A planning LLM, using constrained decoding under a fixed schema, decides which streams, filters, fusion strategy, output fields and answer format a question needs.
  3. That plan is compiled into SQL and executed.
  4. Only the retrieved evidence goes to the answer-generating LLM.

Because the model input stays small regardless of recording length, accuracy no longer drops as audio gets longer. The approach also handles speaker-conditioned and temporal reasoning zero-shot, without hand-written SQL or task-specific prompts. This suggests that explicit retrieval planning is a practical route to scalable, multimodal audio reasoning.

Key points

  • Plan compilation: Each selected stream becomes a Common Table Expression (CTE). A hybrid LLM–rule-based generator combines these into one SQL query, and streams are merged with a simple nearest-neighbor temporal join on timestamps. Retrieval itself is deliberately keyword-based.
  • Input length: Gemini reads over 115k tokens for 60 minutes of speech, while PlanRAG-Audio cuts the LLM input to roughly 1k tokens. Without the framework, 17.92% of Gemini's diarization outputs for 10–540-minute audio could not be parsed.
  • Single-modality reasoning with Gemini:
    • Speaker counting accuracy rose from 14.20% to 69.40%.
    • Spearman correlation for sound event ordering rose from 0.30 to 0.68.
  • Speaker-constrained MCQA:
    • Gemini's abstention accuracy reached 94.90% with PlanRAG-Audio.
    • Qwen with PlanRAG-Audio reached 67.59% QA accuracy and 82.20% abstention accuracy.
  • Limitations:
    • Results depend on the accuracy of the upstream perception models.
    • Preprocessing adds computational cost, which may limit real-time use.
    • Gemini API instability constrained parts of the evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Information retrieval & search · 473 worksSpeech & spoken language · 480 works

Cite

Cite as Masao Someki, Chien-yu Huang, Siddhant Arora, Samuele Cornell, Markus Müller, Nathan Susanj, Rupak Vignesh Swaminathan, Grant Strimel, Jing Liu and Shinji Watanabe. 2026. PlanRAG-Audio: Planning and Retrieval Augmented Generation for Long-form Audio Understanding. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1304

@inproceedings{someki2026planrag,
  title = {{PlanRAG-Audio: Planning and Retrieval Augmented Generation for Long-form Audio Understanding}},
  author = {Masao Someki, Chien-yu Huang, Siddhant Arora, Samuele Cornell, Markus Müller, Nathan Susanj, Rupak Vignesh Swaminathan, Grant Strimel, Jing Liu and Shinji Watanabe},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1304},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1304}
}

References

40 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jerome Abdelnour, Giampiero Salvi, and Jean Rouat. 2019. Clear: A dataset for compositional language and elementary acoustic reasoning. find
  2. Jérôme Abdelnour, Jean Rouat, and Giampiero Salvi. 2023. Naaqa: A neural architecture for acoustic question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  3. Orevaoghene Ahia, Martijn Bartelds, Kabir Ahuja, Hila Gonen, Valentin Hofmann, Siddhant Arora, Shuyue Stella Li, Vishal Puttagunta, Mofetoluwa Adeyemi, Charishma Buchireddy, Ben Walls, Noah Bennett, Shinji Watanabe, Noah A. Smith, Yulia Tsvetkov, and Sachin Kumar. 2025. Blab: Brutally long audio ben… — [arXiv:2505.03054](https://arxiv.org/abs/2505.03054) find
  4. Siddhant Arora, Haidar Khan, Kai Sun, Xin Luna Dong, Sajal Choudhary, Seungwhan Moon, Xinyuan Zhang, Adithya Sagar, Surya Teja Appini, Kaushik Patnaik, Sanat Sharma, Shinji Watanabe, Anuj Kumar, Ahmed Aly, Yue Liu, Florian Metze, and Zhaojiang Lin. 2025. Stream rag: Instant and accurate spoken dialo… — [arXiv:2510.02044](https://arxiv.org/abs/2510.02044) find
  5. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations.
  6. Carlos Busso, Reza Lotfian, Kusha Sridhar, Ali N. Salman, Wei-Cheng Lin, Lucas Goncalves, Srinivas Parthasarathy, Abinay Reddy Naini, Seong-Gyun Leem, Luz Martinez-Lucas, Huang-Cheng Chou, and Pravin Mote. 2025. The msp-podcast corpus. Preprint, arXiv:2509.09791. — [arXiv:2509.09791](https://arxiv.org/abs/2509.09791) find
  7. Yifu Chen, Shengpeng Ji, Haoxiao Wang, Ziqing Wang, Siyu Chen, Jinzheng He, Jin Xu, and Zhou Zhao. 2025. WavRAG: Audio-integrated retrieval augmented generation for spoken dialogue models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape…
  8. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Édouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. Preprint, arXiv:2410.00037.
  9. Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. find
  10. Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). find
  11. Gemini Team Google. 2023. et al. gemini: a family of highly capable multimodal models.
  12. Alexander Johnson, Peter Plantinga, Pheobe Sun, Swaroop Gadiyaram, Abenezer Girma, and Ahmad Emami. 2024. Efficient SQA from Long Audio Contexts: A Policy-driven Approach. In Interspeech 2024. find
  13. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  14. Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley. 2020. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing. find
  15. Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. 2024. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. In Proceedings of the 41st International Conference on Machine Learning (ICML). find
  16. Myeonghwa Lee, Seonho An, and {Min Soo} Kim. 2024. Planrag: A plan-then-retrieval augmented generation for generative large language models as decision makers. find
  17. NAACL 2024. Association for Computational Linguistics (ACL). find
  18. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th Internation…
  19. Chyi-Jiunn Lin, Guan-Ting Lin, Yung-Sung Chuang, Wei-Lun Wu, Shang-Wen Li, Abdelrahman Mohamed, Hung-Yi Lee, and Lin-Shan Lee. 2024. Speechdpr: End-to-end spoken passage retrieval for open-domain spoken question answering. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and… find
  20. Guan-Ting Lin, Yung-Sung Chuang, Ho-Lam Chung, Shu wen Yang, Hsuan-Jui Chen, Shuyan Annie Dong, Shang-Wen Li, Abdelrahman Mohamed, Hung yi Lee, and Lin shan Lee. 2022. Dual: Discrete spoken unit adaptive learning for textless spoken question answering. In Interspeech 2022. find
  21. Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. 2022. Clothoaqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO). find
  22. Alexander H. Liu, Andy Ehrenberg, Andy Lo, Clément Denoix, Corentin Barreau, Guillaume Lample, Jean-Malo Delignon, Khyathi Raghavi Chandu, Patrick von Platen, Pavankumar Reddy Muddireddy, Sanchit Gandhi, Soham Ghosh, Srijan Mishra, Thomas Foubert, Abhinav Rastogi, Adam Yang, Albert Q. Jiang, Alexand… find
  23. Heng Liu, Siru Jiang, Fangyun Duan, Yongzhe Lyu, Xiu-song Wang, Hanlin Ge, and Chao Liang. 2025b. Ca-dencerag: Context-aware and dependency-enhanced retrieval augmented generation for holistic video understanding. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPR… find
  24. Yanming Liu, Xinyue Peng, Xuhong Zhang, Weihao Liu, Jianwei Yin, Jiannan Cao, and Tianyu Du. 2024. RA-ISF: Learning to answer and understand from retrieval augmentation via iterative self-feedback. In Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational… find
  25. Leander Melroy Maben, Gayathri Ganesh Lakshmy, Srijith Radhakrishnan, Siddhant Arora, and Shinji Watanabe. 2025. Aura: Agent for understanding, reasoning, and automated tool use in voice-driven tasks. Preprint, arXiv:2506.23049. — [arXiv:2506.23049](https://arxiv.org/abs/2506.23049) find
  26. I. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, W. Post, Dennis Reidsma, and P. Wellner. 2005. The ami meeting corpus. In Proceedings of Measuring Behavior 2005, 5th International… find
  27. Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. 2016. Metrics for polyphonic sound event detection. Applied Sciences, 6(6):162. find
  28. Do June Min, Karel Mundnich, Andy Lapastora, Erfan Soltanmohammadi, Srikanth Ronanki, and Kyu Han. 2025. Speech retrieval-augmented generation without automatic speech recognition. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). find
  29. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). find
  30. Abhirama Subramanyam Penamakuri, Kiran Chhatre, and Akshat Jain. 2025. Audiopedia: Audio qa with knowledge. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). find
  31. Archiki Prasad, Trung Bui, Seunghyun Yoon, Hanieh Deilamsalehy, Franck Dernoncourt, and Mohit Bansal. 2023. MeetingQA: Extractive question-answering on meeting transcripts. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association… find
  32. Sina Semnani, Violet Yao, Heidi Zhang, and Monica Lam. 2023. WikiChat: Stopping the hallucination of large language model chatbots by few-shot grounding on Wikipedia. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics. find
  33. Natarajan Balaji Shankar, Alexander Johnson, Christina Chance, Hariram Veeramani, and Abeer Alwan. 2024. Coraal qa: A dataset and framework for open domain spontaneous speech question answering from long audio files. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal… find
  34. Parthasaarathy Sudarsanam and Tuomas Virtanen. 2023. Attention-based methods for audio question answering. In 31st European Signal Processing Conference (EUSIPCO). EUSIPCO. find
  35. Jinchuan Tian, William Chen, Yifan Peng, Jiatong Shi, Siddhant Arora, Shikhar Bharadwaj, Takashi Maekaku, Yusuke Shinohara, Keita Goto, Xiang Yue, Huck Yang, and Shinji Watanabe. 2025. OpusLM: A Family of Open Unified Speech Language Models. In Interspeech 2025. find
  36. Prakhar Verma, Sukruta Prakash Midigeshi, Gaurav Sinha, Arno Solin, Nagarajan Natarajan, and Amit Sharma. 2025. Plan$^\\ast$RAG: Efficient test-time planning for retrieval augmented generation. In Workshop on Reasoning and Planning for Large Language Models. find
  37. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proceedings of the 59th Annual…
  38. Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, and Chao Zhang. 2025. Salmonn-omni: A standalone speech llm without codec injection for full-duplex conversation. Preprint, arXiv:2505.17060. — [arXiv:2505.17060](https://arxiv.org/abs/2505.17060) find
  39. Zihan Zhao, Yiyang Jiang, Heyang Liu, Yu Wang, and Yanfeng Wang. 2024. Librisqa: A novel dataset and framework for spoken question answering with large language models. IEEE Transactions on Artificial Intelligence. find
  40. Ge Zhu, Jordan Darefsky, and Zhiyao Duan. 2024. Cacophony: An improved contrastive audio-text model. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/someki-2026-planrag-audio-planning-retrieval-augmented/