Elamur Research⌕
☰
home / papers / 2026 / Augmenting LLM Reasoning with Dynamic Notes Writing for Complex MultiHop QA

Augmenting LLM Reasoning with Dynamic Notes Writing for Complex MultiHop QA

Summary

Iterative retrieval-augmented generation helps with multi-hop questions, but each retrieval round adds more documents to the prompt. Much of that material is noisy or redundant. It can overflow the context window, raise inference cost, and distract the model from planning its next step. Earlier compression approaches mostly target single-round retrieval, need finetuning, or scale poorly across many iterations.

The authors propose NotesWriting. At every retrieval step, a separate, smaller note-taking LLM reads each retrieved Wikipedia document and writes short notes relevant to the current query. It skips content that is unrelated or was already captured. The main model then reasons over these aggregated notes instead of the raw documents. The module does not depend on any particular framework, and the authors plug it into IRCoT, FLARE and ReAct. The ReAct variant, called ReNAct, works best.

Across two LLMs and four multi-hop QA benchmarks, the method improves accuracy by 15.6 percentage points on average. It also produces sharper search queries and cleaner reasoning traces. This matters because the model can effectively take in more evidence without retraining.

Key points

  • NotesWriting uses a dedicated smaller model (LM_notes) that extracts query-relevant notes from each of the top-k retrieved documents. The notes are merged and replace the raw documents in the main model's context.
  • The note-taking prompt asks for a YES/NO flag followed by the extracted context. Irrelevant, unanswerable or already-extracted content is marked NO.
  • The method was tested with three iterative RAG frameworks (IRCoT, FLARE, ReAct), two LLMs and four multi-hop QA datasets. It gave an average gain of 15.6 percentage points.
  • ReNAct (ReAct + NotesWriting) shows the largest gains. The authors attribute this to more precise search queries and therefore better retrieved documents.
  • Answer correctness is judged by GPT-4 against ground truth. Reasoning chains are rated on a 0–5 scale for step efficiency, redundancy, and clarity/coherence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4uyke9dmgz56

@inproceedings{maheshwary2026augmenting,
  title = {{Augmenting LLM Reasoning with Dynamic Notes Writing for Complex MultiHop QA}},
  author = {Maheshwary, Rishabh and Hashemi, Masoud and Mahajan, Khyati and Malay, Shiva Krishna Reddy and Mudumba, Sai Rajeswar and Madhusudhan, Sathwik Tejaswi and Gella, Spandana and Yadav, Vikas},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4uyke9dmgz56},
  url = {https://doi.org/10.63317/4uyke9dmgz56}
}

References

46 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohamed Aghzal, Erion Plaku, Gregory J Stein, and Ziyu Yao. 2025. A survey on large language models for automated planning. arXiv preprint arXiv:2502.12435. — [arXiv:2502.12435](https://arxiv.org/abs/2502.12435) find
  2. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511. https://www.mediawiki.org/wiki/API:Search ↗ our card
  3. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learn… find
  4. Yanan Chen, Ali Pesaranghader, Tanmana Sadhu, and Dong Hoon Yi. 2024. Can we rely on llm agents to draft long-horizon plans? let’s take travelplanner as an example. arXiv preprint arXiv:2408.06318. — [arXiv:2408.06318](https://arxiv.org/abs/2408.06318) find
  5. Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2022. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257–273. find
  6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  7. Jingsheng Gao, Linxu Li, Weiyuan Li, Yuzhuo Fu, and Bin Dai. 2024. Smartrag: Jointly learn rag-related tasks from the environment feedback. arXiv preprint arXiv:2410.18141. — [arXiv:2410.18141](https://arxiv.org/abs/2410.18141) find
  8. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654. find our card
  9. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. find our card
  10. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1–43. find
  11. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1–38. find our card
  12. Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983. find
  13. Zhouyu Jiang, Mengshu Sun, Lei Liang, and Zhiqiang Zhang. 2025. Retrieve, summarize, plan: Advancing multi-hop question answering with an iterative approach. find
  14. Ziyan Jiang, Xueguang Ma, and Wenhu Chen. 2024. Longrag: Enhancing retrieval-augmented generation with long-context llms. arXiv preprint arXiv:2406.15319. — [arXiv:2406.15319](https://arxiv.org/abs/2406.15319) find
  15. Ehsan Kamalloo, Nouha Dziri, Charles LA Clarke, and Davood Rafiei. 2023. Evaluating open-domain question answering in the era of large language models. arXiv preprint arXiv:2305.06984. find our card
  16. Greg Kamradt. 2023. Needle in a haystack-pressure testing llms. Github Repository, page 28. find
  17. Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, Kentaro Inui, et al. 2023. Real-time qa: What’s the answer right now? Advances in neural information processing systems, 36:49025–49043. find
  18. Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. 2024. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. arXiv preprint arXiv:2409.12941. find
  19. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626. find
  20. Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien MR Arnold, Vincent Perot, Siddharth Dalmia, et al. 2024a. Can long-context language models subsume retrieval, rag, sql, and more? arXiv preprint arXiv:2406.13121. find
  21. Myeonghwa Lee, Seonho An, and Min-Soo Kim. 2024b. Planrag: A plan-then-retrieval augmented generation for generative large language models as decision makers. arXiv preprint arXiv:2406.12430. find
  22. Quinn Leng, Jacob Portes, Sam Havens, Matei Zaharia, and Michael Carbin. 2024. Long context rag performance of large language models. arXiv preprint arXiv:2411.03538. — [arXiv:2411.03538](https://arxiv.org/abs/2411.03538) find
  23. Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2020. How context affects language models’ factual predictions. arXiv preprint arXiv:2005.04611. find our card
  24. Revanth Gangi Reddy, Sagnik Mukherjee, Jeonghwan Kim, Zhenhailong Wang, Dilek Hakkani-Tur, and Heng Ji. 2024. Infogent: An agent-based framework for web information aggregation. arXiv preprint arXiv:2410.19054. find
  25. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023a. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR. find our card
  26. Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023b. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652. find our card
  27. Yixuan Tang and Yi Yang. 2024. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries. arXiv preprint arXiv:2401.15391. find
  28. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509. find our card
  29. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533. find our card
  30. Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. 2024. How easily do irrelevant inputs skew the responses of large language models? arXiv preprint arXiv:2404.03302. find
  31. Jian Xie, Kexun Zhang, Jiangjie Chen, Siyu Yuan, Kai Zhang, Yikai Zhang, Lei Li, and Yanghua Xiao. 2024. Revealing the barriers of language agents in planning. arXiv preprint arXiv:2410.12409. find
  32. Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2023a. Recomp: Improving retrieval-augmented lms with compression and selective augmentation. arXiv preprint arXiv:2310.04408. find
  33. Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023b. Retrieval meets long context large language models. In The Twelfth International Conference on Learning Representations. find our card
  34. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369… find our card
  35. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR). find our card
  36. Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. 2023. Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.09210. find
  37. Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. 2024. Inference scaling for long-context retrieval augmented generation. arXiv preprint arXiv:2410.04343. — [arXiv:2410.04343](https://arxiv.org/abs/2410.04343) find
  38. Qin Zhang, Shangsi Chen, Dongkuan Xu, Qingqing Cao, Xiaojun Chen, Trevor Cohn, and Meng Fang. 2022. A survey for efficient open domain question answering. arXiv preprint arXiv:2211.07886. find
  39. Andrew Zhu, Alyssa Hwang, Liam Dugan, and Chris Callison-Burch. 2024. FanOutQA: A multi-hop, multi-document question answering benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 18–37, Bangko… find
  40. Examples comparing ReNAct with baselines Model Setting Benchmark Fanout-QA Frames Hotpot-QA MultiHop-RAG GPT-4o-mini Baseline Fanout-QA Frames Hotpot-QA MultiHop-RAG NotesWriting Fanout-QA Frames Hotpot-QA MultiHop-RAG LLaMA-3.1-70B Baseline Fanout-QA Frames Hotpot-QA MultiHop-RAG NotesWriting Table… find
  41. (Result 3) Demographics of the Supreme Court of the United States - The most recent four Supreme Court justices are: Ketanji Brown Jackson (appointed in 2022), Amy Coney Barrett (appointed in 2020), Brett Kavanaugh (appointed in 2018), and Neil Gorsuch (appointed in 2017). find
  42. Thought: The most recent four Supreme Court justices are Amy Coney Barrett, Ketanji Brown Jackson, Neil Gorsuch, and Brett Kavanaugh. Next, I need to find out which law schools they attended. I will start by searching for the law school attended by Amy Coney Barrett. Action: search[Amy Coney Barrett… find
  43. Thought: Amy Coney Barrett attended Notre Dame Law School. Next, I need to find out which law school Ketanji Brown Jackson attended. Action: search[Ketanji Brown Jackson; Which law school did Ketanji Brown Jackson attend?] Observation: (Result 1) Ketanji Brown Jackson attended Harvard Law School, wh… find
  44. (Result 2) Joe Biden Supreme Court candidates - Ketanji Brown Jackson attended Harvard University for her law degree. (Result 3) Demographics of the Supreme Court of the United States - Ketanji Brown Jackson attended Harvard Law School. Thought: Ketanji Brown Jackson attended Harvard Law School. Now… find
  45. Thought: Neil Gorsuch also attended Harvard Law School. Finally, I need to find out which law school Brett Kavanaugh attended. Action: search[Brett Kavanaugh; Which law school did Brett Kavanaugh attend?] Observation: (Result 1) Brett Kavanaugh - Kavanaugh graduated from Yale Law School with a Juris… find
  46. Thought: Brett Kavanaugh graduated from Yale Law School. Now I have gathered the law schools attended by the most recent four Supreme Court justices: Amy Coney Barrett attended Notre Dame Law School, Ketanji Brown Jackson and Neil Gorsuch both attended Harvard Law School, and Brett Kavanaugh attende… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/maheshwary-2026-augmenting-llm-reasoning-dynamic-notes/