Elamur Research⌕
☰
home / papers / 2026 / Domain-Specific Data Generation Framework for RAG Adaptation

Domain-Specific Data Generation Framework for RAG Adaptation

Paper · 2026 · Chris Xing Tian, Weihao Xie, Zhen Chen et al.

Summary

Off-the-shelf RAG pipelines often perform poorly in specialised domains. Adapting their components, such as the embedding model or the generator LLM, requires domain-grounded training data, which is rarely available. The authors present RAGen, a data-centric framework that turns raw domain documents into question–answer–context (QAC) triples for adapting several RAG components at once. The pipeline works in three stages:

  • It splits documents into semantic chunks and extracts concepts from each chunk with ChatGPT-4o.
  • It merges these into document-level concepts by embedding them with OpenAI Ada and clustering with K-means.
  • For each concept, it retrieves evidence from multiple chunks and generates questions at different Bloom's Taxonomy levels, each paired with answers and context variants that include distractors.

The resulting data serves two purposes: training embedding models with a contrastive objective and adapting Qwen2.5 LLMs through supervised training. Across three domains, RAGen data improves both retrieval and answer quality compared with datasets built using AutoRAG and LlamaIndex. Training with distractors makes the generator more robust to noisy retrieval. This matters for organisations that run smaller local models over private, changing knowledge bases.

Key points

  • Pipeline components:
    • Chunking uses the llamaindex chunker.
    • The number of document-level concepts K is a manually set hyperparameter.
    • Each question comes with four context types: fully supportive, partially supportive, irrelevant and misleading.
  • Evaluation setup: experiments cover three domains (PPFS, TradePolicy, BusinessAI). Retrieval is compared against Vanilla, AutoRAG and LlamaIndex baselines, with results averaged over 3 runs.
  • Retrieval: the fine-tuned embedding model trained on RAGen data scored highest on every reported retrieval metric in all three domains.
  • Generation:
    • Qwen2.5-1.5B/3B Instruct models fine-tuned on RAGen data beat the baselines on both ROUGE-L and BERT-F1 in all settings.
    • RAGen's ROUGE-L ranged from about 0.34 to 0.40, while the best baseline scores ranged from about 0.27 to 0.36.
  • Distractor training: on PPFS, Qwen2.5-3B was tested with real top-3 retrieval. Adding distractors during training raised ROUGE-L from 0.3143 to 0.4074 and BERT-F1 from 0.8957 to 0.9121.
  • Limitations:
    • The pipeline handles text-only input.
    • Errors from early stages can propagate to later ones.
    • The evaluation includes no human study.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓City University of Hong KongHong Kong · China45 works · 3,014 citations🔬Peng Cheng LaboratoryShenzhen · China38 works · 2,504 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

Cite

Cite as Chris Xing Tian, Weihao Xie, Zhen Chen, Hui Liu, Zhengyuan Yi, Haoliang Li, Shiqi Wang and Siwei Ma. 2026. Domain-Specific Data Generation Framework for RAG Adaptation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.960

@inproceedings{tian2026domain,
  title = {{Domain-Specific Data Generation Framework for RAG Adaptation}},
  author = {Chris Xing Tian, Weihao Xie, Zhen Chen, Hui Liu, Zhengyuan Yi, Haoliang Li, Shiqi Wang and Siwei Ma},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.960},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.960}
}

References

40 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations.
  2. BAAI. 2024. Bge large en v1.5 model. https://huggingface.co/BAAI/bge-large-en-v1.5. ↗
  3. BAAI. 2025. Flag embedding. https://github.com/ FlagOpen/FlagEmbedding. ↗
  4. Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. 2024. Seven failure points when engineering a retrieval augmented generation system. In Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI, pages 194– 199. find
  5. Jian Chen, Peilin Zhou, Yining Hua, Yingxin Loh, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. 2024a. Fintextqa: A dataset for long-form financial question answering. arXiv preprint arXiv:2405.09980. find
  6. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024b. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216.
  7. Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024c. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754–17762.
  8. Filippo Chiarello, Vito Giordano, Irene Spada, Simone Barandoni, and Gualtiero Fantoni. 2024. Future applications of generative large language models: A data-driven case study on chatgpt. Technovation, 133:103002. find
  9. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers.
  10. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130. find
  11. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians,…
  12. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  13. Seonjeong Hwang, Yunsu Kim, and Gary Geunbae Lee. 2024. Explainable multi-hop question generation: An end-to-end approach without intermediate question labeling. arXiv preprint arXiv:2404.00571. find
  14. Shayekh Bin Islam, Md Asib Rahman, KSM Hossain, Enamul Hoque, Shafiq Joty, and Md Rizwan Parvez. 2024. Open-rag: Enhanced retrieval-augmented reasoning with open-source large language models. arXiv preprint arXiv:2410.01782. find
  15. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In EMNLP (1), pages 6769–6781.
  16. Dongkyu Kim, Byoungwook Kim, Donggeon Han, and Matouš Eibich. 2024. Autorag: Automated framework for optimization of retrieval augmented generation pipeline. Preprint, arXiv:2410.20878. — [arXiv:2410.20878](https://arxiv.org/abs/2410.20878) find
  17. David R Krathwohl. 2002. A revision of bloom’s taxonomy: An overview. Theory into practice, 41(4):212– 218. find
  18. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–…
  19. Hui Liu, Wenya Wang, Hao Sun, Chris Xing Tian, Chenqi Kong, Xin Dong, and Haoliang Li. 2025. Unraveling the mechanics of learning-based demonstration selection for in-context learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),…
  20. LlamaIndex. 2025. Llamaindex datasetgenerator. https://developers.llamaindex.ai/python/ framework-api-reference/evaluation/ dataset_generation/. ↗
  21. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511. find
  22. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
  23. OpenAI. 2025a. Openai chatgpt4o. //platform.openai.com/docs/models/ chatgpt-4o-latest. https: find
  24. OpenAI. 2025b. Openai embedding api. https://platform.openai.com/docs/models/ text-embedding-ada-002. ↗
  25. Crystal Qian, Michael Xieyang Liu, Emily Reif, Grady Simon, Nada Hussein, Nathan Clement, James Wexler, Carrie J Cai, Michael Terry, and Minsuk Kahng. 2024. The evolution of llm adoption in industry data curation practices. arXiv preprint arXiv:2412.16089. — [arXiv:2412.16089](https://arxiv.org/abs/2412.16089) find
  26. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. Qwen2.5 Technical Report. arXiv preprint. ArXiv:2… find
  27. Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Manning. 2024. Raptor: Recursive abstractive processing for tree-organized retrieval. In The Twelfth International Conference on Learning Representations.
  28. Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. 2023. Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering. Transactions of the Association for Computational Linguistics,…
  29. Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592.
  30. Asahi Ushio, Fernando Alva-Manchego, and Jose Camacho-Collados. 2023. An empirical comparison of lm-based question and answer generation methods. arXiv preprint arXiv:2305.17002. find
  31. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.
  32. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368.
  33. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. Preprint, arXiv:2309.07597.
  34. Xiang Yue, Xinliang Frederick Zhang, Ziyu Yao, Simon Lin, and Huan Sun. 2021. Cliniqg4qa: Generating diverse questions for domain adaptation of clinical question answering. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 580–587. IEEE. find
  35. Mingtian Zhang, Shawn Lan, Peter Hayes, and David Barber. 2024a. Mafin: Enhancing black-box embeddings with model augmented fine-tuning. arXiv preprint arXiv:2402.12177. — [arXiv:2402.12177](https://arxiv.org/abs/2402.12177) find
  36. Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. 2024b. Soaring from 4k to 400k: Extending llm’s context with activation beacon. arXiv preprint arXiv:2401.03462, 2(3):5.
  37. Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian-Yun Nie. 2023. Retrieve anything to augment large language models. Preprint, arXiv:2310.07554. — [arXiv:2310.07554](https://arxiv.org/abs/2310.07554) find
  38. Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E Gonzalez. 2024c. Raft: Adapting language model to domain specific rag. In First Conference on Language Modeling.
  39. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372.
  40. Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, and 1 others. 2024. Rageval: Scenario specific rag evaluation dataset generation framework. arXiv preprint arXiv:2408.01262.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tian-2026-domain-specific-data-generation-framework/