Elamur Research⌕
☰
home / papers / 2025 / Aligning Language Models for Icelandic Legal Text Summarization

Aligning Language Models for Icelandic Legal Text Summarization

Paper · 2025 · Þórir Hrafn Harðarson, Hrafn Loftsson and Stefán Ólafsson

Summary

The paper asks whether preference-based alignment helps language models write summaries of Icelandic Supreme Court rulings that legal professionals find acceptable, compared with plain supervised fine-tuning. The authors compare GPT-SW3-1.3B, which was pre-trained on Nordic languages, with Llama2-7B and with Ice-Llama2, a Llama2 variant further pre-trained on a random 10% sample of the Icelandic Gigaword Corpus.

Training runs in three stages:

  • Domain pre-training: continued pre-training on long rulings.
  • Instruction fine-tuning: supervised fine-tuning on ruling–summary pairs.
  • Preference training: DPO and RLHF on pairwise comparisons built from the best-ROUGE model's outputs.

Legal experts ranked the generated summaries and scored each one separately for legal soundness and for Icelandic quality. Preference training raised legal accuracy over supervised fine-tuning, although not consistently. It did not noticeably improve the Icelandic itself, where the language-specific pre-training of GPT-SW3 mattered more. ROUGE scores often disagreed with expert judgments. No model reached human-expert level. The work shows that low-resource, high-stakes domains need human evaluation and a solid language-specific base model.

Key points

  • Data source: rulings from haestirettur.is, released as the thorirhrafn/domar_data dataset on Hugging Face.
  • Two splits:
    • R holds long rulings only: 5,677 rows, split 5,077 / 300 / 300.
    • RS holds ruling–summary pairs that fit the context window: 2,613 rows, split 2,013 / 300 / 300.
  • Training details: domain pre-training packed text into 512-token blocks. Summarization fine-tuning fed one ruling per step and accumulated gradients over 8 sequences before each update.
  • Evaluation design:
    • A primary legal expert ranked outputs for 25 diverse rulings.
    • Two more experts ranked 5 of those cases to check agreement.
    • Each summary received two separate 1–5 scores, one for legal quality and one for Icelandic quality.
  • Limitations: the expert found contradictory statements within single summaries, for example dismissing a case and then denying its dismissal. The summary set (~2,600 rows) is tiny compared with the ~120,000 rows used in earlier RLHF summarization work.

Interesting ideas

  • I1. Scoring each output separately for domain correctness and for language fluency showed that alignment improved one but not the other, which a single score would have hidden.
  • I2. Splitting the corpus by length lets over-long documents still feed unsupervised domain pre-training, while only pairs that fit the window go to the supervised task.
  • I3. The authors treat the mismatch between ROUGE and expert preference as a finding and suggest bringing human judgment in earlier, before choosing which model generates the training data.
  • I4. Having a few extra experts re-rank a subset of cases gives a cheap agreement check on subjective judgments.
  • I5. The qualitative review caught sentence-level contradictions that overlap metrics cannot detect.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluate generated or parsed output on separate axes, such as structural correctness and surface well-formedness, so a gain on one axis cannot mask stagnation on the other.
  • I2 (learn): Texts too long or too messy for gold or silver annotation can still feed unsupervised stages, such as lexicon or frequency statistics, rather than being discarded.
  • I3 (learn): Before letting one parser or teacher tier generate silver data, check its outputs against a small human-judged sample, not only against an automatic score.
  • I4 (answer): When grading open answers to QA on fables or tales, double-annotate a small subset to measure how reliable the grading itself is.
  • I5 (back): After generating a summary from the meaning graph, re-parse it and check its frames for mutually exclusive states of the same entity, flagging contradictions explainably.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksLaw · 77 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSummarization · 232 worksLegal AI · 134 works

Cite

Cite as Þórir Hrafn Harðarson, Hrafn Loftsson and Stefán Ólafsson. 2025. Aligning Language Models for Icelandic Legal Text Summarization. Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hardharson2025aligning,
  title = {{Aligning Language Models for Icelandic Legal Text Summarization}},
  author = {THórir Hrafn Hardharson, Hrafn Loftsson and Stefán Ólafsson},
  booktitle = {Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025)},
  year = {2025},
  url = {https://aclanthology.org/2025.nodalida-1.25/}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Starkaður Barkarson, Steinþór Steingrímsson, and Hildur Hafsteinsdóttir. 2022. Evolving Large Text Corpora: Four Versions of the Icelandic Gigaword Corpus. In Proceedings of the Language Resources and Evaluation Conference, page 11, Marseille, France. find
  2. Amelia Glaese, Nat McAleese, Maja Tr˛ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soňa Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. 2022. Improving alignment of dialogue agents via targeted human judgements. ArXiv.
  3. Andrew Blair-Stanek, Nils Holzenberger, and Benjamin Van Durme. 2023. Can GPT-3 Perform Statutory Reasoning? In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL ’23, page 22–31. Association for Computing Machinery.
  4. Tammy Pettinato Oltz. 2023. Chatgpt, professor of law. University of Illinois Journal of Law, Technology & Policy, page 207. ↗
  5. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations.
  6. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22.
  7. Deepali Jain, Malaya Dutta Borah, and Anupam Biswas. 2021. Summarization of legal documents: Where are we now and the way forward. Computer Science Review, 40:100388. ↗
  8. Eva Hrönn Jónsdóttir. 2023. Helmingur hefur velt fyrir sér að skipta um starfsvettvang. Lögmannablaðið, 03/23:16–18. find
  9. Daniel Martin Katz, Corinna Coupette, Janis Beckedorf, and Dirk Hartung. 2020. Complex societies and the growth of the law. Scientific Reports, 10(1). ↗
  10. Giuseppe Pisano, Alessia Fidelangeli, Federico Galli, Andrea Loreggia, Riccardo Rovatti, Piera Santin, and Giovanni Sartor. 2024. Summarization of tax rulings in the PRODIGIT projec. i-lex, 17(1):1–26. find
  11. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459– 9474. Curran Associates, Inc.
  12. Alec Radford and Karthik Narasimhan. 2018. Improving Language Understanding by Generative Pre-Training. find
  13. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. ↗
  14. Daniele Licari, Praveen Bushipaka, Gabriele Marino, Giovanni Comandé, and Tommaso Cucinotta. 2023. Legal holding extraction from italian case documents using italian-legal-bert text summarization. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL ’23, page 148–156, New York, NY, USA. Association for Computing Machinery. find
  15. Marijn Schraagen, Floris Bex, Nick Van De Luijtgaar-den, and Daniël Prijs. 2022. Abstractive Summarization of Dutch Court Verdicts Using Sequence-to-sequence Models. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 76–87, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  16. Daniele Licari and Giovanni Comandè. 2024. ITALIAN-LEGAL-BERT models for improving natural language processing tasks in the Italian legal domain. Computer Law & Security Review, 52:105908. ↗
  17. Vésteinn Snæbjarnarson, Haukur Barri Símonarson, Pétur Orri Ragnarsson, Svanhvít Lilja Ingólfsdóttir, Haukur Jónsson, Vilhjalmur Thorsteinsson, and Hafsteinn Einarsson. 2022. A Warm Start and a Clean Crawled Corpus - A Recipe for Good Language Models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4356–4366, Marseille, France. European Language Resources Association. find
  18. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  19. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv.
  20. Steinþór Steingrímsson, Sigrún Helgadóttir, Eiríkur Rögnvaldsson, Starkaður Barkarson, and Jón Guðnason. 2018. Risamálheild: A Very Large Icelandic Text Corpus. In Proceedings of LREC 2018, Myazaki, Japan. find
  21. Marion Nickum and Pascale Desrumaux. 2023. Burnout among lawyers: effects of workload, latitude and mediation via engagement and over-engagement. Psychiatry, Psychology and Law, 30(3):349–361. ↗
  22. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc. ↗
  23. Anna Nikulásdóttir, Jón Guðnason, Anton Karl Ingason, Hrafn Loftsson, Eiríkur Rögnvaldsson, Einar Freyr Sigurðsson, and Steinþór Steingrímsson. 2020. Language Technology Programme for Icelandic 2019–2023. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3414–3422, Marseille, France. European Language Resources Association. ↗
  24. Peter M. Tiersma. 1999. Legal Language, 1 edition. University of Chicago Press. 2023. The Nordic Pile: A 1.2TB Nordic Dataset for Language Modeling. ArXiv. find
  25. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. ArXiv, abs/2307.09288. ↗
  26. Dietrich Trautmann, Alina Petrova, and Frank Schilder. 2022. Legal Prompt Engineering for Multilingual Legal Judgement Prediction. ArXiv. ↗
  27. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2024. Zephyr: Direct Distillation of LM Alignment. In Conference on Language Modeling. ↗
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
  29. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗
  30. Fangyi Yu, Lee Quartey, and Frank Schilder. 2022. Legal Prompting: Teaching a Language Model to Think Like a Lawyer. ArXiv. Joey Öhman, Severine Verlinden, Ariel Ekgren, Amaru Cuba Gyllensten, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, and Magnus Sahlgren. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dharson-2025-aligning-language-models-icelandic-legal/