Elamur Research⌕
☰
home / papers / 2026 / Is Micro Domain-Adaptive Pre-Training Effective for Real-World Operations? Multi-Step Evaluation Reveals Potential and Bottlenecks

Is Micro Domain-Adaptive Pre-Training Effective for Real-World Operations? Multi-Step Evaluation Reveals Potential and Bottlenecks

Paper · 2026 · Masaya Tsunokake, Yuta Koreeda, Terufumi Morishita et al.

Summary

Enterprises need LLMs that can work with narrow, proprietary knowledge tied to specific operations ("micro domains"). Earlier work showed that micro domain-adaptive pre-training (mDAPT) helps on multiple-choice questions, but its value for open-ended generation was untested. The authors adapted Qwen2.5-72B-Instruct to Hitachi's JP1 IT product documentation. Their mDAPT combined continual pre-training (CPT) on manuals, mixed with general data, and supervised fine-tuning (SFT) on QA pairs synthesized from the same manuals. They tested the models on real technical support questions.

To find where answering fails, they split it into three stages: eliciting facts, reasoning over them, and composing a long-form answer. They then supplied ideal ("oracle") outputs for each stage and measured the resulting gains. They also probed memorization separately.

mDAPT fixed the base model's inability to recall domain facts, but overall success remained low because reasoning and composing were still weak. This points to reasoning ability as the main lever for making such models usable in practice.

Key points

  • The setup used JP1 (e.g., JP1/AJS3) manuals, with llm-jp-corpus-v2 mixed in during training. Including general-domain data slightly hurt fact memorization but limited catastrophic forgetting of answer-composing skill, a net benefit.
  • The mDAPT model's answer success rate (ASR) without oracles was only about 39%. With GPT-4o, supplying oracle elicitation and reasoning pushed ASR above 90%.
  • SFT on top of CPT helped models move from merely memorizing facts to actually using them, and it preserved composing ability.
  • In expert review of 20 support questions, the 3-CPT-epoch mDAPT model produced more useful answers than RAG and GPT-4o. On 10 questions, adding oracle conclusions raised expert scores, confirming reasoning as a bottleneck.
  • Full-parameter CPT and SFT required 24 H100 GPUs. The authors list lighter methods such as LoRA and mDAPT on reasoning models as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Masaya Tsunokake, Yuta Koreeda, Terufumi Morishita, Koichi Nagatsuka, Hikaru Tomonari and Yasuhiro Sogawa. 2026. Is Micro Domain-Adaptive Pre-Training Effective for Real-World Operations? Multi-Step Evaluation Reveals Potential and Bottlenecks. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.22

@inproceedings{tsunokake2026is,
  title = {{Is Micro Domain-Adaptive Pre-Training Effective for Real-World Operations? Multi-Step Evaluation Reveals Potential and Bottlenecks}},
  author = {Masaya Tsunokake, Yuta Koreeda, Terufumi Morishita, Koichi Nagatsuka, Hikaru Tomonari and Yasuhiro Sogawa},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.22},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.22}
}

References

33 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  2. Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. 2025. How do large language models acquire factual knowledge during pretraining? In Proceedings of the 38th International Conference on Neural Information Processing Systems (NeurIPS 2024), pages 60… find
  3. Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. 2024a. Instruction pre-training: Language models are supervised multitask learners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2529–2550. language models. Computing Re…
  4. Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024b. Adapting large language models via reading comprehension. In The Twelfth International Conference on Learning Representations. find
  5. Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Lin, Wen-tau Yih, and Srini Iyer. 2024. Instruction-tuned language models are better knowledge learners. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo… find
  6. Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre F. T. Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, and Michael Desa. 2024. SaulLM-7B: A pioneering large language model for law. Computing Research Repository, arXiv:2403.03883. find
  7. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. DeepSeek-R1: Incentivizing reasoning capability in ll…
  8. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-augmented generation for large language models: A survey. Computing Research Repository, arXiv:2312.10997.
  9. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi… find
  10. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360. find
  11. Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, Tianjiao Zhao, Amit Panalkar, Wei Cheng, Haoyu Wang, Yanchi Liu, Zhengzhang Chen, Haifeng Chen, Chris White, Quanquan Gu, and 2 others. 2023. Domain specialization as the… find
  12. Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. 2025a. ProRL: Prolonged reinforcement learning expands reasoning boundaries in large language models. Computing Research Repository, arXiv:2505.24864. find
  13. Zhaowei Liu, Xin Guo, Fangqi Lou, Lingfeng Zeng, Jinyi Niu, Zixuan Wang, Jiajie Xu, Weige Cai, Ziwei Yang, Xueqian Zhao, Chao Li, Sheng Xu, Dezhi Chen, Yun Chen, Zuo Bai, and Liwen Zhang. 2025b. Fin-R1: A large language model for financial reasoning through reinforcement learning. Computing Research… — [arXiv:2503.16252](https://arxiv.org/abs/2503.16252) find
  14. Hitachi. 2025a. Integrated operations management JP1. Accessed at 2025/07/30. find
  15. Hitachi. 2025b. Manuals - Middleware - JP1. Accessed at 2025/07/30. find
  16. Hitachi. 2025c. Manuals - Middleware - JP1 - Job scheduler：JP1/Automatic Job Management System 3. Accessed at 2025/10/19. find
  17. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations.
  18. Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. 2025. m1: Unleash the potential of test-time scaling for medical reasoning with large Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rockt… find
  19. LLM-jp corpus v2. Accessed at Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  20. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. Computing Research Repository, arXiv:2501.19393.
  21. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. GPT-4o system card.…
  22. OpenAI. 2023. GPT-4 technical report. Computing Research Repository, arXiv:2303.08774.
  23. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Computing Research Repository, ar… find
  24. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory optimizations toward training trillion parameter models. Computing Research Repository, arXiv:1910.02054. find
  25. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. In Proceedings of the 37th Conference on Neural Informatio…
  26. Ingo Ziegler, Abdullatif Köksal, Desmond Elliott, and Hinrich Schütze. 2024. CRAFT your dataset: Task-specific synthetic dataset generation through corpus retrieval and augmentation. Computing Research Repository, arXiv:2409.02098.
  27. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, and 12 others. 2… — Towards Expert-Level Medical Question Answering with Large Language Models
  28. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. ♫ MuSiQue: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554. find
  29. Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, Yihan Cao, Hui Ren, Xiang Li, Xiaoxiao Li, and Yuyin Zhou. 2025. MedReason: Eliciting factual medical reasoning steps in llms via knowledge graphs. Computing Research Repository,… — [arXiv:2504.00993](https://arxiv.org/abs/2504.00993) find
  30. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. BloombergGPT: A large language model for finance. Computing Research Repository, arXiv:2303.17564.
  31. Yawen Xue, Masaya Tsunokake, Yuta Koreeda, Ekant Muljibhai Amin, Takashi Sumiyoshi, and Yasuhiro Sogawa. 2025. Agent fine-tuning through distillation for domain-specific LLMs in microdomains. Computing Research Repository, arXiv:2510.00482. — [arXiv:2510.00482](https://arxiv.org/abs/2510.00482) find
  32. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
  33. Siyun Zhao, Yuqing Yang, Zilong Wang, Zhiyuan He, Luna K. Qiu, and Lili Qiu. 2024. Retrieval augmented generation (RAG) and beyond: A comprehensive survey on how to make your LLMs use external data more wisely. Computing Research Repository, arXiv:2409.14924. — [arXiv:2409.14924](https://arxiv.org/abs/2409.14924) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tsunokake-2026-is-micro-domain-adaptive-pre/