Elamur Research⌕
☰
home / papers / 2026 / DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimization

DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimization

Paper · 2026 · Haiyang Shen, Hang Yan, Zhongshi Xing et al.

Summary

Because their training data mostly comes from broad, general sources like Wikipedia, dense retrievers have trouble with specialized corpora. Domain texts use unusual terminology, leave reasoning steps implicit, and spread evidence across passages. In RAG pipelines, retrievers also receive LLM-generated sub-queries that may be only partially complete. DRAGON addresses this through synthetic training data. It models how documents, queries, answers and supporting "clues" (source sentences) relate to each other. An entity-centric graph links sentences across documents for multi-hop questions, and randomized rephrasing rules vary logical complexity and clue completeness.

The authors also release DRAGONBENCH, which pairs questions with sentence-level citations and per-question scoring rubrics. The rubrics support a new criteria-based score generation (CSG) metric, which proved steadier than plain LLM-as-a-Judge. Six MTEB retrievers fine-tuned on DRAGON data improved across domains and transferred to unseen corpora. Plugging them into vanilla, planning-based and iterative RAG raised end-to-end accuracy, showing that retriever-side data synthesis is a useful complement to tuning the LLM itself.

Key points

  • DRAGONBENCH: 8 domain-specific document collections across 4 fields. Questions vary in hop count and answerability (full or partial), and each comes with sentence-level citations and its own scoring rubric.
  • Training setup: Six retrievers from the MTEB leaderboard (33M–611M parameters, 512–8192 context length) were fine-tuned with contrastive learning and ANCE hard-negative sampling.
  • In-domain gains: Retrieval improved on four large corpora: Hearthstone Wiki, Zelda Wiki, Drugs.com and Mayo Clinic. Ablations attribute much of the gain to the complex reformulations and the clue-completeness data, not only to in-domain exposure.
  • Cross-domain transfer: A retriever trained only on Zelda data improved on four other corpora (Stanford, UC Berkeley, Cyotek, Notion).
  • End-to-end RAG and limitations: The tuned retrievers improved results in vanilla RAG, FLARE-style iterative RAG and RR-style planning RAG, for both single-hop and multi-hop queries. The approach is limited to factual QA with deterministic answers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓Chinese University of Hong KongHong Kong · China126 works · 14,073 citations🎓Sun Yat-sen UniversityGuangzhou · China53 works · 9,185 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Haiyang Shen, Hang Yan, Zhongshi Xing, Mugeng Liu, Yue Li, Zhiyang Chen, Yuxiang Wang, Jiuzheng Wang and Yun Ma. 2026. DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimization. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.56

@inproceedings{shen2026dragon,
  title = {{DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimization}},
  author = {Haiyang Shen, Hang Yan, Zhongshi Xing, Mugeng Liu, Yue Li, Zhiyang Chen, Yuxiang Wang, Jiuzheng Wang and Yun Ma},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.56},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.56}
}

References

44 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alibaba-NLP. Gte multilingual https://huggingface.co/Alibaba-NLP/ gte-multilingual-base. ↗
  2. Hangfeng He, Hongming Zhang, and Dan Roth. 2023. Rethinking with retrieval: Faithful large language model inference. Preprint, arXiv:2301.00303. find
  3. Hearthstone Wiki Contributors. 2025. Hearthstone wiki. https://hearthstone.fandom.com/wiki/ Hearthstone_Wiki. ↗
  4. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Comm…
  5. Yucheng Hu and Yuxing Lu. 2024. Rag and rau: A survey on retrieval-augmented language model in natural language processing. arXiv preprint arXiv:2404.19543. — [arXiv:2404.19543](https://arxiv.org/abs/2404.19543) find
  6. Naoya Inoue, Pontus Stenetorp, and Kentaro Inui. 2020. R4C: A benchmark for evaluating RC systems to get the right answer for the right reason. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6740–6750, Online. Association for Computational Linguisti… find
  7. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research.
  8. Aryan Jadon, Avinash Patil, and Shashank Kumar. 2025. Enhancing domain-specific retrieval-augmented generation: Synthetic data generation and evaluation using reasoning models. Preprint, arXiv:2502.15854. find
  9. Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2024. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992, Singapore. Association fo…
  10. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver…
  11. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361.
  12. Minsang Kim and Seung Jun Baek. 2025. Syntriever: How to train your retriever with synthetic data from LLMs. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2523–2539, Albuquerque, New Mexico. Association for Computational Linguistics. find
  13. Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo… find
  14. Abhinand Krishnappa. 2025. Medembed. https://github.com/abhinand5/MedEmbed. Accessed: 2025-02-12. ↗
  15. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural…
  16. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. NV-embed: Improved techniques for training LLMs as generalist embedding models. In The Thirteenth International Conference on Learning Representations.
  17. Xueguang Ma, Xi Victoria Lin, Barlas Oguz, Jimmy Lin, Wen tau Yih, and Xilun Chen. 2025. Drama: Diverse augmentation from large language models to smaller dense retrievers. Preprint, arXiv:2502.18460.
  18. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… find
  19. Mayo Clinic. 2025. Mayo clinic. mayoclinic.org/. https://www. find
  20. Luke Merrick. 2024. Embedding and clustering your data can improve contrastive pretraining. Preprint, arXiv:2407.18887. — [arXiv:2407.18887](https://arxiv.org/abs/2407.18887) find
  21. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistic…
  22. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th…
  23. Notion. 2023. Notion help center. notion.com/help. https://www. find
  24. Jing-Cheng Pang, Heng-Bo Fan, Pengyuan Wang, Jia-Hao Xiao, Nan Tang, Si-Hang Yang, Chengxing Jia, Sheng-Jun Huang, and Yang Yu. 2024. Empowering language models with active inquiry for deeper understanding. arXiv preprint arXiv:2402.03719. — [arXiv:2402.03719](https://arxiv.org/abs/2402.03719) find
  25. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating large language models to master 16000+ real…
  26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  27. Sergeyzh. Rubert-tiny-turbo. https://huggingface.co/sergeyzh/rubert-tiny-turbo. ↗
  28. Haiyang SHEN, Yue Li, Desong Meng, Dongqi Cai, Sheng Qi, Li Zhang, Mengwei Xu, and Yun Ma. 2025. Shortcutsbench: A large-scale real-world benchmark for API-based agents. In The Thirteenth International Conference on Learning Representations. find
  29. Stanford University. 2025. Stanford university admissions. https://admission.stanford.edu/. ↗
  30. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554. find
  31. University of California, Berkeley. 2025. University of california, berkeley admissions. https://admissions.berkeley.edu/. ↗
  32. Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, and Jianghong Ma. 2025a. Rocketeval: Efficient automated LLM evaluation via grading checklist. In The Thirteenth International Conference on Learning Representations. find
  33. Zhepei Wei, Wei-Lin Chen, and Yu Meng. 2025b. Instructrag: Instructing retrieval-augmented generation via self-synthesized rationales. Preprint, arXiv:2406.13629. — [arXiv:2406.13629](https://arxiv.org/abs/2406.13629) find
  34. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations.
  35. Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu JIANG, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nicolas SCHEFFER, Yue Liu, Nirav Shah, Rakesh Wanga, Anu… find
  36. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
  37. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
  38. Fan Yin, Zifeng Wang, I-Hung Hsu, Jun Yan, Ke Jiang, Yanfei Chen, Jindong Gu, Long T. Le, Kai-Wei Chang, Chen-Yu Lee, Hamid Palangi, and Tomas Pfister. 2025. Magnet: Multi-turn tool-use data synthesis and distillation via graph translation. Preprint, arXiv:2503.07826. find
  39. Zelda Wiki Contributors. 2025. Zelda wiki. https://zelda.fandom.com/wiki/Main_Page. ↗
  40. Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023. Agenttuning: Enabling generalized agent abilities for llms. Preprint, arXiv:2310.12823.
  41. Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. 2025. Jasper and stella: distillation of sota embedding models. Preprint, arXiv:2412.19048. — [arXiv:2412.19048](https://arxiv.org/abs/2412.19048) find
  42. Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng, Tulika Awalgaonkar, Rithesh Murthy, Eric Hu, Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, and Caiming Xi… — xLAM: A Family of Large Action Models to Empower AI Agent Systems
  43. Siyun Zhao, Yuqing Yang, Zilong Wang, Zhiyuan He, Luna K Qiu, and Lili Qiu. 2024. Retrieval augmented generation (rag) and beyond: A comprehensive survey on how to make your llms use external data more wisely. arXiv preprint arXiv:2409.14924. — [arXiv:2409.14924](https://arxiv.org/abs/2409.14924) find
  44. Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. 2024. Megapairs: Massive data synthesis for universal multimodal retrieval. Preprint, arXiv:2412.14475.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2026-dragon-domain-specific-robust-automatic/