Elamur Research⌕
☰
home / papers / 2026 / Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training

Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training

Paper · 2026 · Lukas Borggren, Jenny Kunz and Marco Kuhlmann

Summary

The paper asks whether general-purpose LLMs can be specialised for Swedish journalism, a niche that combines a lower-resource language with an understudied domain, without training a model from scratch. The authors cleaned 21.7M Bonnier News articles (1991–2026) into BonCorpus. They then ran continued pre-training (CPT) on Ministral models at two sizes, comparing full fine-tuning (FFT) with LoRA. Code, English and Swedish data from Common Corpus were mixed in as experience replay. Because few suitable test sets existed, they also built BonEval, a benchmark of six editorial tasks.

CPT improved generation and factual knowledge but not discriminative tasks such as classification and entity extraction. It only helped when replay data was included. LoRA was competitive with FFT and often better. Adding an "instruct vector" without further training helped LoRA models but not FFT ones. The general Swedish benchmark EuroEval mostly missed these in-domain gains, which shows that domain-specific evaluation is needed. The work offers a practical recipe for publishers and others adapting models to a specific language and domain.

Key points

  • BonCorpus: Preprocessing included normalisation, regex cleanup, fastText language-ID and metadata filters, plus exact and fuzzy (MinHash LSH + Levenshtein) deduplication. It cut the collection by almost a third, leaving 14.6M articles and 8.5B tokens.
  • BonEval tasks: The six tasks are Headline, Lead, Summary (generative); Topic, with 19 labels extending the IPTC Media Topics, and Entity, which is salient NER (discriminative); and Quiz, multiple-choice over 15 categories (knowledge).
  • BonEval data: Test articles come only from 2022 onward and are removed from the training data. Headline, Lead, Topic and Entity have 8,192 examples each, Summary has 100 and Quiz has 11,960.
  • Replay mixtures: Every combination of the three replay types was tested, with each included type making up 10% of the tokens. Leaving replay out produced negative results.
  • Native vs translated tests: On EuroEval, CPT caused large drops on the LLM-translated MMLU and HellaSwag. On natively Swedish tasks such as SwedishFacts, the LoRA model roughly matched or slightly beat the base model. The data, code and models are proprietary and not released.

Interesting ideas

  • I1. To deduplicate, the authors link near-duplicate pairs into a graph, take its connected components and keep only the longest document per group, which collapses whole clusters of reposted text rather than just pairs.
  • I2. Test items are drawn from a recent time window and then removed from the training corpus, a simple guard against overlap with the base model's own pre-training data.
  • I3. The gap between translated and native benchmarks suggests generic or machine-translated tests can hide, or even contradict, real progress in a target domain.
  • I4. Salient NER keeps only the persons, organisations and places that are central to an article, rather than every entity mentioned, turning extraction into a relevance judgement.
  • I5. Testing every combination of replay data types shows which kinds of old data actually protect existing skills when adapting to a new domain.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Cluster near-identical sentences in silver-data pools this way before training or scoring, so repeated templates neither inflate the training signal nor leak into evaluation.
  • I2 (learn): Build held-out evaluation sets for parsers and readers from newer, explicitly excluded documents, so gains reflect generalisation rather than memorised text.
  • I3 (answer): Keep our own native test suites (school tests, word problems, fable QA) as the main yardstick, and treat generic benchmarks only as a secondary signal.
  • I4 (world): Add a salience ranking over entities in the world of a text, so answers and summaries focus on central participants and skip incidental mentions.
  • I5 (tree): When adapting gparse to a new genre or a converted UD dialect, mix in portions of the original treebanks and ablate the mixtures to measure forgetting in LAS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Lukas Borggren, Jenny Kunz and Marco Kuhlmann. 2026. Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training. arXiv:2608.30609.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2608.30609 · arXiv 2608.30609

@misc{borggren2026reading,
  title = {{Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training}},
  author = {Lukas Borggren, Jenny Kunz and Marco Kuhlmann},
  year = {2026},
  doi = {10.48550/arxiv.2608.30609},
  eprint = {2608.30609},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2608.30609}
}

References

85 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. 2024a. Instruction Pre-Training: Language Models are Supervised Multitask Learners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2529–2550, Miami, Florida, United States. Association for Computational Linguistics.
  2. AI Sweden. 2024. AI-Sweden-Models/Llama-3-8B. Model repository. Accessed: 2026-06-16. find
  3. Aleksandrs Berdicevskis, Gerlof Bouma, Robin Kurtz, Felix Morger, Joey Öhman, Yvonne Adesam, Lars Borin, Dana Dannélls, Markus Forsberg, Tim Isbister, Anna Lindahl, Martin Malmsten, Faton Rekathati, Magnus Sahlgren, Elena Volodina, Love Börjeson, Simon Hengchen, and Nina Tahmasebi. 2023. Superlim: A Swedish Language Understanding Evaluation Benchmark. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8137–8153, Singapore. Association for Computational Linguistics. find
  4. Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024b. Adapting Large Language Models via Reading Comprehension. In Twelfth International Conference on Learning Representations. find
  5. Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Melo, Dominic Culver, Etienne Malaboeuf, Gabriel Hautreux, Johanne Charpentier, and Michael Desa. 2024. SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain. In Advances in Neural Information Processing Systems, volume 37, pages 129672–129695. find
  6. Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. 2024. LoRA Learns Less and Forgets Less. Transactions on Machine Learning Research. ↗
  7. Javier de la Rosa, Vladislav Mikhailov, Lemei Zhang, Freddy Wetjen, David Samuel, Peng Liu, Rolv-Arild Braaten, Petter Mæhlum, Magnus Breder Birkenes, Andrey Kutuzov, Tita Enstad, Hans Christian Farsethås, Svein Arne Brygfjeld, Jon Atle Gulla, Stephan Oepen, Erik Velldal, Wilfred Østgulen, Lilja Øvrelid, and Aslak Sira Myhre. 2025. The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies, pages 544–560, Tallinn, Estonia. University of Tartu Library. find
  8. Andrei Z. Broder. 1997. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997, pages 21–29. find
  9. Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc’Aurelio Ranzato. 2019. On Tiny Episodic Memories in Continual Learning. Computing Research Repository, arXiv:1902.10486. Version 4. ↗
  10. Jie Chen, Zhipeng Chen, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Yingqian Min, Wayne Xin Zhao, Zhicheng Dou, Jiaxin Mao, Yankai Lin, Ruihua Song, Jun Xu, Xu Chen, Rui Yan, Zhewei Wei, Di Hu, Wenbing Huang, and Ji-Rong Wen. 2025. Towards Effective and Efficient Continual Pre-training of Large Language Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5779–5795, Vienna, Austria. Association for Computational Linguistics.
  11. Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, and Stefano Soatto. 2024. Fewer Truncations Improve Language Modeling. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pages 11030–11048. ↗
  12. Aleksei Dorkin, Taido Purason, Emil Kalbaliyev, Hele-Andra Kuulmets, Marii Ojastu, Mark Fišel, Tanel Alumäe, Eleri Aedmaa, Krister Kruusmaa, and Kairit Sirts. 2026. EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training. Computing Research Repository, arXiv:2603.02041. Version 1. ↗
  13. Pinzhen Chen, Simon Yu, Zhicheng Guo, and Barry Haddow. 2024. Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9706–9726, Miami, Florida, United States. Association for Computational Linguistics. find
  14. Simen Eide. 2024. Schibsted Text Tasks: Introducing New Datasets for Norwegian and Swedish Language Models. Blog post. Accessed: 2026-06-16. find
  15. Ariel Ekgren, Amaru Cuba Gyllensten, Evangelia Gogoulou, Alice Heiman, Severine Verlinden, Joey Öhman, Fredrik Carlsson, and Magnus Sahlgren. 2022. Lessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedish. In Proceedings of the Thirteenth Conference on Language Resource and Evaluation, pages 3509–3518. find
  16. Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. 2023. MEDITRON-70B: Scaling Medical Pre-training for Large Language Models. Computing Research Repository, arXiv:2311.16079. Version 1. ↗
  17. Ariel Ekgren, Amaru Cuba Gyllensten, Felix Stollenwerk, Joey Öhman, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, Judit Casademont, and Magnus Sahlgren. 2024. GPT-SW3: An Autoregressive Language Model for the Scandinavian Languages. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, pages 7886–7900. Models: How to re-warm your model? In Workshop on Efficient Systems for Foundation Models. find
  18. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics. ↗
  19. Ahmed Elhady, Eneko Agirre, and Mikel Artetxe. 2025. Emergent Abilities of Large Language Models under Continued Pre-training for Language Adaptation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32174–32186, Vienna, Austria. Association for Computational Linguistics. find
  20. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. In Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. ↗
  21. Julen Etxaniz, Oscar Sainz, Naiara Perez, Itziar Aldabe, German Rigau, Eneko Agirre, Aitor Ormazabal, Mikel Artetxe, and Aitor Soroa. 2024. Latxa: An Open Language Model and Evaluation Suite for Basque. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14952–14972, Bangkok, Thailand. Association for Computational Linguistics. ↗
  22. Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pásztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, Ido Hakimi, Juan Garcia Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, and 83 others. 2026. Apertus: Democratizing Open and Compliant LLMs for Global Language Environments. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 46877–46955, San Diego, California, United States. Association for Computational Linguistics.
  23. EuroEval. 2026. EuroEval Datasets: Swedish. Dataset documentation. Accessed: 2026-06-16. find
  24. Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities. In First Conference on Language Modeling. ↗
  25. Masanori Hirano and Kentaro Imajo. 2024. Construction of Domain-Specified Japanese Large Language Model for Finance Through Continual Pre-Training. In 2024 16th IIAI International Congress on Advanced Applied Informatics, pages 273–279. ↗
  26. Kevin Glocker, Kätriin Kukk, Romina Oji, Marcel Bollmann, Marco Kuhlmann, and Jenny Kunz. 2025. Grow Up and Merge: Scaling Strategies for Efficient Language Adaptation. Computing Research Repository, arXiv:2512.10772. Version 1. ↗
  27. Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, Yanning Chen, and Zhipeng Wang. 2025. Liger-Kernel: Efficient Triton Kernels for LLM Training. In Championing Open-source Development in ML Workshop. ↗
  28. Evangelia Gogoulou, Timothée Lesort, Magnus Boman, and Joakim Nivre. 2024. Continual Learning Under Language Shift. In Text, Speech, and Dialogue: 27th International Conference, pages 71–84.
  29. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In Tenth International Conference on Learning Representations.
  30. Haiyang Guo, Fanhu Zeng, Fei Zhu, Jiayi Wang, Xukai Wang, Jingang Zhou, Hongbo Zhao, Wenzhuo Liu, Shijie Ma, Xu-Yao Zhang, and Cheng-Lin Liu. 2025a. Continual Learning for Generative AI: From LLMs to MLLMs and Beyond. Computing Research Repository, arXiv:2506.13045. Version 4. ↗
  31. Yiduo Guo, Jie Fu, Huishuai Zhang, and Dongyan Zhao. 2025b. Efficient Domain Continual pretraining by Mitigating the Stability Gap. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32850–32870, Vienna, Austria. Association for Computational Linguistics. find
  32. Shih-Cheng Huang, Pin-Zu Li, Yu-chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tsai, and Hung-yi Lee. 2024. Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10943–10959, Bangkok, Thailand. Association for Computational Linguistics. ↗
  33. Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. 2023. Continual Pre-Training of Large Language Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timothée Lesort, and Irina Rish. 2024. Simple and Scalable Strategies to Continually Pre-train Large Language Models. Transactions on Machine Learning Research. Mexico City, Mexico. Association for Computational Linguistics. find
  34. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing Models with Task Arithmetic. In Eleventh International Conference on Learning Representations. ↗
  35. Achintya Kundu, Rhui Dih Lee, Laura Wynter, Raghu Kiran Ganti, and Mayank Mishra. 2024. Enhancing Training Efficiency Using Packing with Flash Attention. Computing Research Repository, arXiv:2407.09105. Version 6. ↗
  36. International Press Telecommunications Council. 2026. IPTC NewsCodes: Media Topics. Controlled vocabulary. Accessed: 2026-06-16. find
  37. Jenny Kunz. 2026a. A Diagnostic Benchmark for Sweden-Related Factual Knowledge. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, pages 5328–5334. find
  38. Ishan Jindal, Chandana Badrinath, Pranjal Bharti, Lakkidi Vinay, and Sachin Dev Sharma. 2024. Balancing Continuous Pre-Training and Instruction Fine-Tuning: Optimizing Instruction-Following in LLMs. Computing Research Repository, arXiv:2410.10739. Version 1. ↗
  39. Jenny Kunz. 2026b. Preferences for Idiomatic Language are Acquired Slowly — and Forgotten Quickly: A Case Study on Swedish. Transactions of the Association for Computational Linguistics, 14:1266– 1285. ↗
  40. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of Tricks for Efficient Text Classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Association for Computational Linguistics. find
  41. Jenny Kunz, Iben Nyholm Debess, and Annika Simonsen. 2026. Family Matters: Language Transfer and Merging for Adapting Small LLMs to Faroese. Computing Research Repository, arXiv:2510.00810. Version 2. ↗
  42. Hele-Andra Kuulmets and Mark Fishel. 2023. Translated Benchmarks Can Be Misleading: the Case of Estonian Question Answering. In Proceedings of the 24th Nordic Conference on Computational Linguistics, pages 710–716, Tórshavn, Faroe Islands. University of Tartu Library. find
  43. Dmytro Kalpakchi and Johan Boye. 2023. SweCTRL-Mini: a data-transparent Transformer-based large language model for controllable text generation in Swedish. Computing Research Repository, arXiv:2304.13994. Version 3. ↗
  44. Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5848– 5864, Bangkok, Thailand. Association for Computational Linguistics.
  45. Zixuan Ke, Yifei Ming, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. 2025. Demystifying Domain-adaptive Post-training for Financial LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31033– 31059, Suzhou, China. Association for Computational Linguistics. find
  46. Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023. Continual Pre-training of Language Models. In Eleventh International Conference on Learning Representations. ↗
  47. Pierre-Carl Langlais, Pavel Chizhov, Catherine Arnett, Carlos Rosas Hinostroza, Mattia Nee, Eliot Krzysztof Jones, Irène Girard, David Mach, Anastasia Stasenko, and Ivan P. Yamshchikov. 2026. Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training. In Fourteenth International Conference on Learning Representations. ↗
  48. Zixuan Ke, Yijia Shao, Haowei Lin, Hu Xu, Lei Shu, and Bing Liu. 2022. Adapting a Language Model While Preserving its General Knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10177– 10188, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  49. Jin-woo Lee, Junhwa Choi, Bongkyu Hwang, Jinho Choo, Bogun Kim, Jeongseon Yi, Joonseok Lee, DongYoung Jung, Jaeseon Park, Kyoungwon Park, and Suk-hoon Jung. 2026. SCALE: Upscaled Continual Learning of Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2026, pages 41006–41020, San Diego, California, United States. Association for Computational Linguistics. find
  50. Sanghoon Kim, Dahyun Kim, Chanjun Park, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim. 2024. SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pages 23–35, Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating Training Data Makes Language Models Better. In Proceedings of the 60th Annual Meeting of the Association Julius Monsen and Arne Jönsson. 2021. A Method for Building Non-English Corpora for Abstractive Text Summarization. In CLARIN Annual Conference Proceedings, pages 82–85. for Computational Linguistics (Volume 1: Long Papers), pages 8424–8445, Dublin, Ireland. Association for Computational Linguistics. ↗
  51. David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. 2004. RCV1: A New Benchmark Collection for Text Categorization Research. Journal of Machine Learning Research, 5:361–397. ↗
  52. Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023. Scaling Data-Constrained Language Models. In Advances in Neural Information Processing Systems, volume 36, pages 50358–50376. find
  53. Jiangtong Li, Yuxuan Bian, Guoxuan Wang, Yang Lei, Dawei Cheng, Zhijun Ding, and Changjun Jiang. 2023. CFGPT: Chinese Financial Assistant with Large Language Model. Computing Research Repository, arXiv:2309.10654. Version 2. ↗
  54. Arijit Nag, Soumen Chakrabarti, Animesh Mukherjee, and Niloy Ganguly. 2025. Efficient Continual Pre-training of LLMs for Low-resource Languages. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track), pages 304–317, Albuquerque, New Mexico, United States. Association for Computational Linguistics. find
  55. Miao Li, Ming-Bin Chen, Bo Tang, ShengbinHou ShengbinHou, Pengyu Wang, Haiying Deng, Zhiyu Li, Feiyu Xiong, Keming Mao, Cheng Peng, and Yi Luo. 2024. NewsBench: A Systematic Evaluation Framework for Assessing Editorial Capabilities of Large Language Models in Chinese Journalism. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9993–10014, Bangkok, Thailand. Association for Computational Linguistics. ↗
  56. Nikhil Shivakumar Nayak, Krishnateja Killamsetty, Ligong Han, Abhishek Bhandwaldar, Prateek Chanda, Kai Xu, Oleg Silkin, Mustafa Eyceoz, Hao Wang, Aldo Pareja, and Akash Srivastava. 2026. Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning. In Fourteenth International Conference on Learning Representations. ↗
  57. Pin-Jie Lin, Rishab Balasubramanian, Fengyuan Liu, Nikhil Kandpal, and Tu Vu. 2025. Efficient Model Development through Fine-tuning Transfer. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2617– 2636, Suzhou, China. Association for Computational Linguistics. find
  58. Dan Saattrup Nielsen. 2023. ScandEval: A Benchmark for Scandinavian Natural Language Processing. In Proceedings of the 24th Nordic Conference on Computational Linguistics, pages 185–201, Tórshavn, Faroe Islands. University of Tartu Library. ↗
  59. Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, Alexandre Sablayrolles, Amélie Héliou, Amos You, Andy Ehrenberg, Andy Lo, Anton Eliseev, Antonia Calvi, Avinash Sooriyarachchi, Baptiste Bout, and 101 others. 2026. Ministral 3. Computing Research Repository, arXiv:2601.08584. Version 1. ↗
  60. Dan Saattrup Nielsen, Kenneth Enevoldsen, and Peter Schneider-Kamp. 2025. Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies, pages 561–572, Tallinn, Estonia. University of Tartu Library. find
  61. Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In Seventh International Conference on Learning Representations. find
  62. Sachita Nishal, Charlotte Li, and Nicholas Diakopoulos. 2024. Domain-Specific Evaluation Strategies for AI in Journalism. Computing Research Repository, arXiv:2403.17911. Version 1. ↗
  63. Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Massimo Hong, Yushuai Wu, Mu Qiao, and Zaiqing Nie. 2026. BioMedGPT: An Open Multimodal Large Language Model for BioMedicine. IEEE Journal of Biomedical and Health Informatics, 30(2):981–992. find
  64. Tobias Norlund and Agnes Stenbom. 2021. Building a Swedish Open-Domain Conversational Language Model. In Proceedings of the 23rd Nordic Conference on Computational Linguistics, pages 357–366, Reykjavik, Iceland (Online). Linköping University Electronic Press. ↗
  65. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. ↗
  66. Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Reuse, Don’t Retrain: A Recipe for Continued Pre-training of Language Models. Computing Research Repository, arXiv:2407.07263. Version 1. ↗
  67. Bettina Messmer, Vinko Sabolčec, and Martin Jaggi. 2025. Enhancing Multilingual LLM Pretraining with Model-Based Data Selection. In Advances in Neural Information Processing Systems, volume 38, pages 70037–70073. find
  68. Guilherme Penedo, Hynek Kydlíček, Loubna B. Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems, volume 37, pages 30811–30849.
  69. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, and 4 others. 2025. Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, Vienna, Austria. Association for Computational Linguistics.
  70. Pouya Pezeshkpour and Estevam Hruschka. 2025. Learning Beyond the Surface: How Far Can Continual Pre-Training with LoRA Enhance LLMs’ Domain-Specific Insight Learning? Computing Research Repository. Version 1. ↗
  71. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics. ↗
  72. Shamane Siriwardhana, Mark McQuade, Thomas Gauthier, Lucas Atkins, Fernando Fernandes Neto, Luke Meyers, Anneketh Vij, Tyler Odenthal, Charles Goddard, Mary MacCarthy, and Jacob Solawetz. 2024. Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluation. Computing Research Repository, arXiv:2406.14971. Version 1. ↗
  73. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory optimizations Toward Training Trillion Parameter Models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. find
  74. Eshaan Tanwar, Deepak Nathani, William Yang Wang, and Tanmoy Chakraborty. 2025. Understanding the Effects of Domain Finetuning on LLMs. Computing Research Repository, arXiv:2510.09359. Version 1. ↗
  75. Pablo Rodríguez, Silvia Paniagua Suárez, Pablo Gamallo, and Susana Sotelo Docio. 2025. Continued Pretraining and Interpretability-Based Evaluation for Low-Resource Languages: A Galician Case Study. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4622–4637, Vienna, Austria. Association for Computational Linguistics. find
  76. Atula Tejaswi, Nilesh Gupta, and Eunsol Choi. 2024. Exploring Design Choices for Building Language-Specific LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10485–10500, Miami, Florida, United States. Association for Computational Linguistics. find
  77. David Samuel, Vladislav Mikhailov, Erik Velldal, Lilja Øvrelid, Lucas Georges Gabriel Charpentier, Andrey Kutuzov, and Stephan Oepen. 2025. Small Languages, Big Models: A Study of Continual Training on Languages of Norway. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies, pages 573–608, Tallinn, Estonia. University of Tartu Library. ↗
  78. Zhao Tong, Yimeng Gu, Huidong Liu, Qiang Liu, Shu Wu, Haichao Shi, and Xiao-Yu Zhang. 2025. Generate First, Then Sample: Enhancing Fake News Detection with LLM-Augmented Reinforced Sampling. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24276–24290, Vienna, Austria. Association for Computational Linguistics. find
  79. Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022. Fine-tuned Language Models are Continual Learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6107–6122, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  80. Kentaro Ueda, François Portet, Hirohiko Suwa, and Keiichi Yasumoto. 2026. Merging Continual Pre-training Models for Domain-Specialized LLMs: A Case Study in Finance. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, pages 10114–10129. find
  81. Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. 2024. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. In Advances in Neural Information Processing Systems, volume 37, pages 68658–68685.
  82. Jiajing Wan, Samia Touileb, Lubos Steskal, and Lilja Øvrelid. 2026. Personalizing News Headlines with Retrieval-Augmented Generation. In Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual, pages 55– 67, San Diego, California, United States. Association for Computational Linguistics.
  83. Zhengxiang Shi and Aldo Lipani. 2023. Don’t Stop Pre-training? Make Prompt-based Fine-tuning Powerful Learner. In Advances in Neural Information Processing Systems, volume 36, pages 5827–5849. find
  84. Shumin Wang, Yuexiang Xie, Bolin Ding, Jinyang Gao, and Yanyong Zhang. 2025. Language Adaptation of Large Language Models: An Empirical Study on LLaMA2. In Proceedings of the 31st International Conference on Computational Linguistics, Silo AI. 2024. Viking 7B/13B/33B: Sailing the Nordic Seas of Multilinguality. Blog post. Accessed: 2026-06-16. pages 7195–7208, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. 11th Baltic Conference on Human Language Technologies, pages 812–825, Tallinn, Estonia. University of Tartu Library.
  85. Zhonghao Wang, Zijia Lu, Bo Jin, and Haiying Deng. 2023. MediaGPT : A Large Language Model For Chinese Media. Computing Research Repository, arXiv:2307.10930. Version 2. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/borggren-2026-reading-news-adapting-large-language/