Elamur Research⌕
☰
home / papers / 2026 / Tracing Multilingual Knowledge Acquisition Dynamics in Domain Adaptation: A Case Study of Biomedical Adaptation

Tracing Multilingual Knowledge Acquisition Dynamics in Domain Adaptation: A Case Study of Biomedical Adaptation

Paper · 2026 · Xin Zhao, Naoki Yoshinaga, Yuma Tsuta et al.

Summary

The paper investigates how multilingual LLMs pick up domain knowledge during continual training, and why that knowledge rarely moves from one language to another. Earlier work often tested models on benchmarks whose facts did not match the training data. To fix this mismatch, the authors build AdaXEval, a pipeline that generates multiple-choice QA items directly from the training corpus. The items probe three abilities: memorization through cloze prompts, intralingual generalization through paraphrased questions, and interlingual generalization through the parallel text in the other language. As a case study, they continually train LLM-jp models on English–Japanese biomedical abstracts from J-STAGE and score saved checkpoints with loss-based answer selection. Accuracy on cloze and paraphrase items grows gradually, while cross-lingual transfer stays weak. The authors explain the in-language gains with a mechanism they call "loss shielding": as the model overfits, losses on wrong options climb faster than the loss on the correct one. Gains stall once overfitting pushes all option losses upward. This matters for designing domain-adaptation data recipes for low-resource languages.

Key points

  • Data: The J-STAGE subset contains 614,444 Japanese and 404,643 English biomedical abstracts, aligned as one-to-one pairs. AdaXEval sampled 10,000 bilingual documents, which yielded 3,236 English and 2,553 Japanese QA items after quality filtering.
  • Pipeline:
    • Sentences are first filtered with NER.
    • Three open-source LLMs then vote on whether each sentence states a fact (combined score > 2 out of 3).
    • GPT-4.1 writes the cloze queries, the paraphrases and three distractors per item.
    • Human evaluation confirmed the quality of the generated data.
  • Training setup: Continual training was augmented with instruction pretraining (rule-based mining plus LLM-generated QA).
  • Overfitting: Perturbation tests show that models overfit easily to fixed token sequences, even when only slight noise is added.
  • Cross-lingual transfer: The authors tested translation and romanization (using cutlet) as extra data. Cross-lingual token overlap within related domains proved important for transfer. Even so, medical translation data raised accuracy by only about 3% while doubling the training cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan90 works · 9,156 citations🔬National Institute of InformaticsTokyo · Japan37 works · 3,247 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Xin Zhao, Naoki Yoshinaga, Yuma Tsuta and Akiko Aizawa. 2026. Tracing Multilingual Knowledge Acquisition Dynamics in Domain Adaptation: A Case Study of Biomedical Adaptation. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.269

@inproceedings{zhao2026tracing,
  title = {{Tracing Multilingual Knowledge Acquisition Dynamics in Domain Adaptation: A Case Study of Biomedical Adaptation}},
  author = {Xin Zhao, Naoki Yoshinaga, Yuma Tsuta and Akiko Aizawa},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.269},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.269}
}

References

34 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Francis Bond and Takayuki Kuribayashi. 2023. The Japanese Wordnet 2.0. In Proceedings of the 12th Global Wordnet Conference, pages 179–186, University of the Basque Country, Donostia - San Sebastian, Basque Country. Global Wordnet Association. find
  2. Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. 2024. How do large language models acquire factual knowledge during pretraining? In Advances in Neural Information Processing Systems, volume 37, pages 60626–60668. Curran Associates, Inc. find
  3. Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. 2024. Journey to the center of the knowledge neurons: discoveries of language-independent knowledge neurons and degenerate knowledge neurons. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth C… find
  4. Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. 2024a. Instruction pre-training: Language models are supervised multitask learners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2529–2550, Miami, Florida, USA. Associat…
  5. Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024b. Adapting large language models via reading comprehension. In The Twelfth International Conference on Learning Representations. find
  6. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained Transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493– 8502, Dublin, Ireland. Association for Computatio… find
  7. DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. Preprint, arXiv:2501.12948.
  8. Yihao Fang, Xianzhi Li, Stephen Thomas, and Xiaodan Zhu. 2023. ChatGPT as data augmentation for compositional generalization: A case study in open intent detection. In Proceedings of the Fifth Workshop on Financial Technology and Natural Language Processing and the Second Multimodal AI For Financial… find
  9. Changjiang Gao, Hongda Hu, Peng Hu, Jiajun Chen, Jixing Li, and Shujian Huang. 2024. Multilingual pre-training and instruction tuning improve cross-lingual knowledge alignment, but only shallowly. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computationa… find
  10. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and Others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  12. Evan Hernandez, Belinda Z Li, and Jacob Andreas. 2024. Inspecting and editing knowledge representations in language models. In Proceedings of the first Conference on Language Modeling. find
  13. Ryosuke Ishigami. 2025. DeepSeek-R1-Distill-Qwen-32B-Japanese. find
  14. Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022a. TemporalWiki: A lifelong benchmark for training and evaluating ever-evolving language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processin… find
  15. Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2022b. Towards continual knowledge learning of language models. In The Tenth International Conference on Learning Representations. find
  16. Evaluation Conference, pages 6704–6710, Marseille, France. European Language Resources Association. find
  17. Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, and Hitoshi Isahara. 2016. ASPEC: Asian scientific paper excerpt corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2204– 2208, Porto…
  18. Junfeng Jiang, Fei Cheng, and Akiko Aizawa. 2024a. Improving referring ability for biomedical language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6444–6457, Miami, Florida, USA. Association for Computational Linguistics. find
  19. Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 319–327, Florence, Italy. Association for Computational Linguistics.
  20. Junfeng Jiang, Jiahao Huang, and Akiko Aizawa. 2025. JMedBench: A benchmark for evaluating Japanese biomedical large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5918–5935, Abu Dhabi, UAE. Association for Computational Linguistics. find
  21. Jingcheng Niu, Andrew Liu, Zining Zhu, and Gerald Penn. 2024. What does the knowledge neuron thesis have to do with knowledge? In The Twelfth International Conference on Learning Representations. find
  22. Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. 2024b. Improving domain adaptation through extended-text reading comprehension. Preprint, arXiv:2401.07284. — [arXiv:2401.07284](https://arxiv.org/abs/2401.07284) find
  23. Mathieu Laï-king and Patrick Paroubek. 2024. Pre-training data selection for biomedical domain adaptation using journal impact metrics. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pages 363–369, Bangkok, Thailand. Association for Computational Linguistics. find
  24. Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Felicia Körner, Ercong Nie, Barbara Plank, François Yvon, and Hinrich Schuetze. 2025. Tracing multilingual factual knowledge acquisition in pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 2121–2146, Suzhou… find
  25. LLM-jp. 2024. LLM-jp: A cross-organizational project for the research and development of fully open Japanese LLMs. Preprint, arXiv:2407.03963. — [arXiv:2407.03963](https://arxiv.org/abs/2407.03963) find
  26. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  27. Soumen Kumar Mondal, Sayambhu Sen, Abhishek Singhania, and Preethi Jyothi. 2025. Language-specific neurons do not facilitate cross-lingual transfer. In The Sixth Workshop on Insights from Negative Results in NLP, pages 46–62, Albuquerque, New Mexico. Association for Computational Linguistics. find
  28. Makoto Morishita, Katsuki Chousa, Jun Suzuki, and Masaaki Nagata. 2022. JParaCrawl v3.0: A large-scale English-Japanese parallel corpus. In Proceedings of the Thirteenth Language Resources and Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander M… find
  29. Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, Philip Mansfield, Blaise Aguera y Arcas, Dale Webste… — Large Language Models Encode Clinical Knowledge
  30. Social Computing Lab. 2023. Mednern-cr-ja (revision 13dbcb6). find
  31. Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. 2022. Finding skill neurons in pre-trained transformer-based language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11132–11152, Abu Dhabi, United Arab Emirates.…
  32. Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijing Xu, Haoqiang Kang, Ziyan Kuang, Chenhan Yuan, Kailai Yang, Zheheng Luo, Tianlin Zhang, Zhiwei Liu, Guojun Xiong, and 15 others. 2024a. FinBen: a holistic financial bench… find
  33. Yong Xie, Karan Aggarwal, and Aitzaz Ahmad. 2024b. Efficient continual pre-training for building domain specific large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10184–10201, Bangkok, Thailand. Association for Computational Linguistics. find
  34. Xin Zhao, Zehui Jiang, and Naoki Yoshinaga. 2025. Neuron empirical gradient: Discovering and quantifying neurons’ global linear controllability. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 21446–21477, Vienna, Austria. Ass… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-tracing-multilingual-knowledge-acquisition-dynamics/