Elamur Research⌕
☰
home / papers / 2026 / Mitigating Structural Knowledge Collapse in Domain-Specific LLMs via Morpheme-Aware KV-Aggregation

Mitigating Structural Knowledge Collapse in Domain-Specific LLMs via Morpheme-Aware KV-Aggregation

Paper · 2026 · Yuxuan Si, Zheqi Lv, Chengxi Zang et al.

Summary

Subword tokenizers such as BPE often split specialized medical and legal terms (e.g., Hydrochlorothiazide) into pieces that do not match their morphemes. The authors argue that attention then fails to reassemble these pieces into one concept, and they call this failure Structural Knowledge Collapse (SKC). Instead of expanding the vocabulary and retraining embeddings, they propose MorphKA, an attention adapter that leaves the tokenizer untouched. In early layers, IMA finds morpheme-like spans (CSUs) using hidden-state similarity and surface cues, then merges their keys and values. In deeper layers, AMRF routes tokens to latent concept centroids and pools K/V. A KL-based fidelity loss keeps attention close to the original model, and a gate limits intervention to fragmented spans. The method was tested on Qwen3-8B and Llama-3.1-8B with medical and legal benchmarks, including PubMedQA and CaseHOLD. MorphKA outperforms vocabulary expansion and dynamic merging baselines, especially on heavily fragmented terms, and loses less general ability on MMLU/GSM8K. It offers a cheap, deployment-compatible way to adapt LLMs to terminology-dense domains.

Key points

  • Accuracy gains: The abstract reports average gains of 3.2–4.6% over the strongest baselines, while the conclusion states 3.9–5.5%. On high-fragmentation examples (p_max ≥ 5), gains reach 7.9–8.7%. Interference on MMLU/GSM8K drops by 18–22, which the paper describes inconsistently as % or percentage points.
  • Efficiency: MorphKA trains about 15.7M parameters, compared with roughly 82M for vocabulary expansion (about 80% fewer). It adds about 1.5GB of training memory on an A100-80GB. Inference runs at 71 tokens/s on Qwen3-8B versus 78 for LoRA SFT, and sequence length and RoPE compatibility are preserved.
  • Ablation (Qwen3-8B):
    • Removing IMA causes the largest domain drop, from 70.55 to 66.95.
    • Removing the fidelity regularizer lowers the GCI score from 83.30 to 77.30.
  • Mechanistic probe: Across 600+ high-fragmentation terms, Top-1 intra-span attention mass rises by 68–72%, and normalized attention entropy (NAE) falls by 24–27%.
  • Limitation: The method depends on well-structured morphology and may work less well for languages with less standardized morphology. The excerpt shows no evaluation outside English medical and legal data, so applicability to Ukrainian is untested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cornell UniversityIthaca · United States120 works · 59,940 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🏢MeituanBeijing · China17 works · 3,769 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMorphology · 405 works

Cite

Cite as Yuxuan Si, Zheqi Lv, Chengxi Zang, Zhengyu Chen and Fei Wu. 2026. Mitigating Structural Knowledge Collapse in Domain-Specific LLMs via Morpheme-Aware KV-Aggregation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1355

@inproceedings{si2026mitigating,
  title = {{Mitigating Structural Knowledge Collapse in Domain-Specific LLMs via Morpheme-Aware KV-Aggregation}},
  author = {Yuxuan Si, Zheqi Lv, Chengxi Zang, Zhengyu Chen and Fei Wu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1355},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1355}
}

References

33 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ehsaneddin Asgari, Yassine El Kheir, and Mohammad Ali Sadraei Javaheri. 2025. Morphbpe: A morpho-aware tokenizer bridging linguistic complexity for efficient llm training across morphologies. Preprint, arXiv:2502.00894. — [arXiv:2502.00894](https://arxiv.org/abs/2502.00894) find
  2. Gunjan Balde, Soumyadeep Roy, Mainack Mondal, and Niloy Ganguly. 2025a. Evaluation of LLMs in medical text summarization: The role of vocabulary adaptation in high OOV settings. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22989–23004, Vienna, Austria. Association fo… find
  3. Gunjan Balde, Soumyadeep Roy, Mainack Mondal, and Niloy Ganguly. 2025b. Evaluation of LLMs in medical text summarization: The role of vocabulary adaptation in high OOV settings. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22989–23004, Vienna, Austria. Association fo… find
  4. Khuyagbaatar Batsuren, Ekaterina Vylomova, Verna Dankers, Tsetsuukhei Delgerbaatar, Omri Uzan, Yuval Pinter, and Gábor Bella. 2024. Evaluating subword tokenization: Alien subword composition and oov generalization challenge. arXiv. find
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv.
  7. Mathias Creutz and Krista Lagus. 2007. Unsupervised models for morpheme segmentation and morphology learning. ACM Transactions on Speech and Language Processing, 4(1):3:1–3:34. find
  8. Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for Chinese LLaMA and Alpaca. arXiv.
  9. Abhimanyu Dubey et al. 2024. The Llama 3 herd of models. arXiv.
  10. Darius Feher, Ivan Vulić, and Benjamin Minixhofer. 2025a. Retrofitting large language models with dynamic tokenization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29866– 29883, Vienna, Austria. Association for Computatio…
  11. Darius Feher, Ivan Vulić, and Benjamin Minixhofer. 2025b. Retrofitting large language models with dynamic tokenization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29866– 29883, Vienna, Austria. Association for Computatio…
  12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In Proceedings of the International Conference on Learning Representations.
  13. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-rank adaptation of large language models. arXiv.
  14. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  15. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
  16. Prakhar Kaushik, Alex Gain, Adam Kortylewski, and Alan Yuille. 2021. Understanding catastrophic forgetting and remembering in continual learning with optimal relevance mapping. arXiv. find
  17. Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023. Continual pre-training of language models. arXiv. find
  18. Anastassia Kornilova and Vladimir Eidelman. 2019. Billsum: A corpus for automatic summarization of US legislation. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 48–56, Hong Kong, China. Association for Computational Linguistics.
  19. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ…
  20. Chengyuan Liu, Yangyang Kang, Shihang Wang, Lizhi Qing, Fubang Zhao, Chao Wu, Changlong Sun, Kun Kuang, and Fei Wu. 2024a. More than catastrophic forgetting: Integrating general capabilities for domain-specific LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Proc… find
  21. Chengyuan Liu, Shihang Wang, Lizhi Qing, Kun Kuang, Yangyang Kang, Changlong Sun, and Fei Wu. 2024b. Gold panning in vocabulary: An adaptive method for vocabulary expansion of domain-specific LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7442–… find
  22. Xin Liu, Baosong Yang, Dayiheng Liu, Haibo Zhang, Weihua Luo, Min Zhang, Haiying Zhang, and Jinsong Su. 2021. Bridging subword gaps in pretrain-finetune paradigm for natural language generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th I… find
  23. Meeting of the Association for Computational Linguistics, pages 1715–1725, Berlin, Germany. Association for Computational Linguistics. find
  24. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NeurIPS 2017).
  26. Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. 2021. Robust fine-tuning of zero-shot models. find
  27. Qihang Yu, Kairui Fu, Zheqi Lv, Shengyu Zhang, Xinhui Wu, Chen Lin, Feng Wei, Bo Zheng, and Fei Wu. 2026. Thinkrec: Thinking-based recommendation via LLM. In WWW, pages 5698–5709. ACM. find
  28. Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. 2021. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the 18th International Conference on Artificial Intelligence and Law, Sao… find
  29. Zheqi Lv, Tianyu Zhan, Wenjie Wang, Xinyu Lin, Shengyu Zhang, Wenqiao Zhang, Jiwei Li, Kun Kuang, and Fei Wu. 2025. Collaboration of large language models and small recommendation models for device-cloud recommendation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data… find
  30. OpenAI. 2023. Gpt-4 technical report. arXiv.
  31. Qiwei Peng, Yekun Chai, and Anders Søgaard. 2025. Understanding subword compositionality of large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 22524–22535, Suzhou, China. Association for Computational Linguistics. find
  32. Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. Mimicking word embeddings using subword RNNs. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 102–112, Copenhagen, Denmark. Association for Computational Linguistics. find
  33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/si-2026-mitigating-structural-knowledge-collapse-domain/