Elamur Research⌕
☰
home / papers / 2026 / Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement

Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement

Paper · 2026 · Aaditya Shukla, Sidney Knowles, Meenakshi Madugula et al.

Summary

Production AI agents in companies tend to degrade after launch, and user complaints rarely feed back into model updates. The authors address this by building a closed-loop "data flywheel" based on the MAPE (Monitor–Analyze–Plan–Execute, with a shared Knowledge base) model from self-adaptive systems. They deployed it around NVInfo AI, NVIDIA's internal Mixture-of-Experts RAG assistant for over 30,000 employees.

Over three months they collected thumbs-up/down ratings and implicit signals such as repeated queries. They then manually traced the negative cases to specific pipeline stages. The two clearest component-level failures were wrong expert routing and faulty query rephrasal. Both were fixed with LoRA fine-tuning through NVIDIA NeMo Microservices.

A fine-tuned Llama 3.1 8B router matched the 70B baseline's accuracy at much lower latency. The rephrasal model, trained on synthetic data, became both more accurate and faster.

The work matters because it offers a reusable blueprint for continuous agent improvement. It also covers practical issues: sparse feedback, privacy rules (PII removal, GDPR/CCPA), and safe staged rollouts.

Key points

  • Feedback data: 1,224 production feedback samples (729 positive, 495 negative). Routing errors were 26/495 (5.25%). Rephrasal errors were about 3.2%, extrapolated from a manual review of 250 samples.
  • Routing fix: Llama 3.1 70B was replaced by a fine-tuned Llama 3.1 8B, trained on 685 deduplicated samples with a 60/40 split.
    • Accuracy stayed at 96%, while latency fell from 0.26 s to 0.08 s (about 70%).
    • For comparison, the untuned 8B model reached only 14% and the prompt-tuned 8B reached 86%.
  • Smaller routers: Fine-tuned Llama 3.2 3B and 1B models both reached 94% routing accuracy.
  • Rephrasal fix: Training used 5,000 synthetic examples generated with Llama 3.1 405B (80/10/10 split).
    • Accuracy rose from 73.8% to 77.5%, a gain of 3.7 points.
    • Latency dropped from 1.9 s to 1.1 s (about 40%).
  • Main limitations: The process depends on slow manual error attribution, users give few ratings, and privacy policies forbid storing full query–response pairs. All training ran on 4× A100 80 GB GPUs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢NvidiaSanta Clara · United States25 works · 5,058 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Aaditya Shukla, Sidney Knowles, Meenakshi Madugula, David Farris, Ryan Angilly, Santiago Pombo, Lu An, Anbang Xu, Abhinav Balasubramanian, Tan Yu, Jiaxiang Ren and Rama Akkiraju. 2026. Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.33

@inproceedings{shukla2026adaptive,
  title = {{Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement}},
  author = {Aaditya Shukla, Sidney Knowles, Meenakshi Madugula, David Farris, Ryan Angilly, Santiago Pombo, Lu An, Anbang Xu, Abhinav Balasubramanian, Tan Yu, Jiaxiang Ren and Rama Akkiraju},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.33},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.33}
}

References

42 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. Arcaini, E. Riccobene, and P. Scandurra. 2015. Modeling and analyzing mape-k feedback loops for self-adaptation. In 2015 IEEE/ACM 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, pages 13–23, Florence, Italy. find
  2. N. Belhaj. 2018. Generic autonomic service management for component-based applications. Ph.D. thesis, Université Paris Saclay (COmUE). Artificial Intelligence [cs.AI]. find
  3. Bitrock. 2024. A comparative analysis of open-source large language models on hugging face. Accessed: 2025-09-09. find
  4. Weiyu Cai, Jiarui Jiang, Fangzhou Wang, Jie Tang, Sunghyun Kim, and Jian Huang. 2025. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering. find
  5. Jui-Chieh Yu Chen, Seungjae Yun, Elias Stengel-Eskin, Tianyi Chen, and Mohit Bansal. 2025. Symbolic mixture-of-experts: Adaptive skill-based routing for heterogeneous reasoning. arXiv preprint arXiv:2503.05641. find
  6. Shizhe Chen, Wenhao Jiang, Bin Lin, James Kwok, and Yaqian Zhang. 2024. Routerdc: Query-based router by dual contrastive learning for assembling large language models. In Advances in Neural Information Processing Systems, volume 37, pages 66305–66328. find
  7. Edward N Coleman, Lorenzo Quarantiello, Zhiqiang Liu, Qiang Yang, Subhabrata Mukherjee, Jose Hurtado, and Vincenzo Lomonaco. 2025. Parameter-efficient continual fine-tuning: A survey. arXiv preprint arXiv:2504.13822. — [arXiv:2504.13822](https://arxiv.org/abs/2504.13822) find
  8. Constellation Research. 2025. Nvidia nemo microservices generally available, aims for ai agent data flywheel. Accessed: 2025-09-09. find
  9. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems, volume 36, pages 10088–10115.
  10. Imen Abdennadher. 2022. Daacs: A decision approach for autonomic computing systems. The Journal of Supercomputing, 78:3883–3904. find
  11. Elias Frick, Tian Li, Chunting Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, Jiantao Jiao, Biao Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024. How to evaluate reward models for rlhf. arXiv preprint arXiv:2410.14872. find
  12. Rama Akkiraju, Anbang Xu, Deb Bora, Tong Yu, Lin An, Vishal Seth, Abhishek Shukla, Pritam Gundecha, Hima Mehta, Ankur Jha, and Piyush Raj. 2024. Facts about building retrieval augmented generation-based chatbots. arXiv preprint arXiv:2407.07858. — [arXiv:2407.07858](https://arxiv.org/abs/2407.07858) find
  13. Joao Gama, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. 2014. A survey on concept drift adaptation. ACM Computing Surveys, 46(4):44. find
  14. Jesper Andersson, Mauro Caporuscio, Marlon D’Angelo, and 1 others. 2023. Architecting decentralized control in large-scale self-adaptive systems. Computing, 105:1849–1882. find
  15. Ming Gao, Xiaoyang Hu, Xi Yin, Jiarui Ruan, Xinyu Pu, and Xiaojun Wan. 2025. Llm-based nlg evaluation: Current status and challenges. Computational Linguistics, pages 1–27.
  16. Omid Gheibi, Danny Weyns, and Fatemeh Quin. 2020. Applying machine learning in self-adaptive systems: A systematic literature review. ACM Transactions on Autonomous and Adaptive Systems, 15(3):Article 9. find
  17. Di Gong, Peng Lu, Zhi Wang, Ming Zhou, and Xiaodong He. 2024. Training agents with weakly supervised feedback from large language models. arXiv preprint arXiv:2411.19547. — [arXiv:2411.19547](https://arxiv.org/abs/2411.19547) find
  18. C.-E. Hrabia, M. Lützenberger, and S. Albayrak. 2018. Towards adaptive multi-robot systems: self-organization and self-adaptation. The Knowledge Engineering Review, 33:E16. find
  19. NVIDIA. 2025c. Overview of nemo microservices. Accessed: 2025-09-09. find
  20. NVIDIA Docs. 2025. About evaluating — nvidia nemo microservices. Accessed: 2025-09-09. find
  21. K. Patel. 2025. Agentic ai for self-healing production lines: Autonomous root cause analysis & correction. Journal of Information Systems Engineering and Management. find
  22. Quotient Blog. 2024. Subject-matter expert language liaison (smell): A framework for aligning llm evaluators to human feedback. find
  23. Edward J Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), volume 1, page 3.
  24. Alexander Ratner, Stephen Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: rapid training data creation with weak supervision. Proceedings of the VLDB Endowment, 11(3):269–282. find
  25. IBM Corporation. 2006. An architectural blueprint for autonomic computing (4th ed.). IBM White Paper. find
  26. A. Romero-Garcés, A. Hidalgo-Paniagua, M. González-García, and A. Bandera. 2022. On managing knowledge for mape-k loops in self-adaptive robotics using a graph-based runtime model. Applied Sciences, 12(17):8583. find
  27. De La Iglesia and Danny Weyns. 2015. Mape-k formal templates to rigorously design behaviors for self-adaptive systems. ACM Transactions on Autonomous and Adaptive Systems, 10(3):Article 15. 31 pages. find
  28. Md Tahmid Rahman Laskar, Saeed Alqahtani, Md Shad Akhtar Bari, Md Rahman, Md Aridul Mamun Khan, Humayun Khan, Ishrat Jahan, Asif Bhuiyan, Ching Wei Tan, Md Rakib Parvez, and Enamul Hoque. 2024. A systematic survey and critical review on evaluating large language models: Challenges, limitations, and…
  29. J. Li, M. Zhang, N. Li, D. Weyns, Z. Jin, and K. Tei. 2024. Generative ai for self-adaptive systems: State of the art and research roadmap. ACM Transactions on Autonomous and Adaptive Systems, 19(3):Article 13. find
  30. Ronghuan Li, Liang He, Qiang Liu, Ziyang Zhang, Hao Yu, Yaqing Ye, Lihua Zhu, and Yixuan Su. 2025. Unirag: Unified query understanding method for retrieval augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), pages 14163–14178. find
  31. Shichao Mao, Yuxin Jiang, Bo Chen, Xiaoyang Li, Peng Wang, Xue Wang, Ping Xie, Fei Huang, Hao Chen, and Nan Zhang. 2024. Rafe: Ranking feedback improves query rewriting for rag. arXiv preprint arXiv:2405.14431. find
  32. B. P. Sanwouo, C. Quinton, and P. Temple. 2025. Breaking the loop: Aware is the new mape-k. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering (FSE Companion ’25), pages 626–630, New York, NY, USA. Association for Computing Machinery. find
  33. Shijie Tan, Shihan Zhuang, Kyle Montgomery, Wei Yang Tang, Adrian Cuadron, Chengliang Wang, Raluca Ada Popa, and Ion Stoica. 2024. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784. find
  34. Unit8. 2024. Road to on-premise llm adoption – part 3. Accessed: 2025-09-09. find
  35. Varun Vats, Muhammad Bilal Nizam, Ming Liu, Zhen Wang, Richard Ho, Manish S. Prasad, Victoria Titterton, Suresh V. Malreddy, Rishabh Aggarwal, Yiming Xu, and Lei Ding. 2024. A survey on human-ai teaming with large pre-trained models. arXiv preprint arXiv:2403.04931. — [arXiv:2403.04931](https://arxiv.org/abs/2403.04931) find
  36. Microsoft Research. 2024. Arena learning: Build data flywheel for llms post-training via simulated chatbot arena. find
  37. NVIDIA. 2025a. Maximize ai agent performance with data flywheels using nvidia nemo microservices. NVIDIA Developer Blog. find
  38. NVIDIA. 2025b. Nemo \| build, monitor, and optimize ai agents. Accessed: 2025-09-09. find
  39. Eric Rutten, Nicolas Marchand, and David Simon. 2017. Feedback control as mape-k loop in autonomic computing. In Rogério de Lemos, David Garlan, Carlo Ghezzi, and Holger Giese, editors, Software Engineering for Self-Adaptive Systems III. Assurances, volume 9640 of Lecture Notes in Computer Science.… find
  40. Aoran Yang, Cheng Chen, and Konstantinos Pitas. 2024. Just rephrase it! uncertainty estimation in closed-source language models via multiple rephrased queries. arXiv preprint arXiv:2405.13907. — [arXiv:2405.13907](https://arxiv.org/abs/2405.13907) find
  41. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Shihan Zhuang, Zhuohan Wu, Yonghao Zhuang, Zi Lin, Zhengxiao Li, Daniel Li, Eric Xing, and Hao Zhang. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623.
  42. Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M. Dai, Quoc V. Le, and James Laudon. 2022. Mixture-of-experts with expert choice routing. In Advances in Neural Information Processing Systems, volume 35, pages 7103– 7114. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shukla-2026-adaptive-data-flywheel-applying-mape/