Elamur Research⌕
☰
home / papers / 2026 / LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics

LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics

Paper · 2026 · Galadrielle Humblot-Renaux, Mohammad N. S. Jahromi, Rohat Bakuri-Jørgensen et al.

Summary

Studying how the Danish Refugee Appeals Board (RAB) judges whether asylum seekers' accounts are believable has so far relied on slow manual coding by legal experts. The authors ask whether off-the-shelf LLMs can do this coding reliably. The task is hard because it is a medium-resource language, uses specialised legal vocabulary, and credibility judgments are often implicit or entangled with risk assessment. They release RAB-Cred, a Danish dataset of RAB decisions. Two refugee-law experts labelled each decision for whether a credibility assessment is present and, if so, whether it is positive or negative, using a jointly developed codebook. The dataset also records annotator confidence and case outcome. They benchmark 21 open-weight LLMs with 30 system-user prompt variants in zero- and few-shot settings, then analyse errors case by case: how consistent mistakes are across models, which classes get confused, how errors relate to human confidence, and how severe they are. LLMs prove useful for low-cost labelling but are inconsistent. Combining several annotators clearly beats any single one, which supports ensemble and human-in-the-loop workflows for legal annotation.

Key points

  • RAB-Cred contains 273 stratified-sampled decisions: 3 few-shot examples, 70 for validation and codebook development, and 200 held-out test cases. Decisions average about 600 words. Data and code are at github.com/glhr/RAB-Cred.
  • The evaluation covers 21 open-weight multilingual models of at most 35B parameters, including EuroLLM 22B and Bielik 11B v3. Prompting strategies include chain-of-thought, metacognitive and few-shot prompting. Prompts are in English while the input texts are in Danish.
  • Prompt choice turned out to matter at least as much as model choice. At the level of individual cases, switching prompts can change predictions more than switching models.
  • An ensemble of the 15 best LLM annotators outperformed every individual model. Two test cases were misclassified by 14 and by all 15 models. On review, an expert judged the LLM answer defensible in both cases.
  • Inference used greedy and constrained decoding (via the outlines library), with one run per configuration. Closed models such as GPT-4o or Claude were excluded because the sensitive legal data required local, offline inference.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CopenhagenCopenhagen · Denmark116 works · 15,292 citations🎓Aalborg UniversityAalborg · Denmark13 works · 743 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksLegal AI · 150 works

Cite

Cite as Galadrielle Humblot-Renaux, Mohammad N. S. Jahromi, Rohat Bakuri-Jørgensen, Marieke Anne Heyl, Asta S. Stage Jarlner, Maria Vlachou, Anna Murphy Høgenhaug, Desmond Elliott, Thomas Gammeltoft-Hansen and Thomas B. Moeslund. 2026. LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics. Proceedings of the 20th Linguistic Annotation Workshop (LAW XX).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.law-main.15

@inproceedings{humblotrenaux2026llms,
  title = {{LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics}},
  author = {Galadrielle Humblot-Renaux, Mohammad N. S. Jahromi, Rohat Bakuri-Jørgensen, Marieke Anne Heyl, Asta S. Stage Jarlner, Maria Vlachou, Anna Murphy Høgenhaug, Desmond Elliott, Thomas Gammeltoft-Hansen and Thomas B. Moeslund},
  booktitle = {Proceedings of the 20th Linguistic Annotation Workshop (LAW XX)},
  year = {2026},
  doi = {10.18653/v1/2026.law-main.15},
  url = {https://doi.org/10.18653/v1/2026.law-main.15}
}

References

45 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 others. 2024. Phi-4 technical report. Preprint, arXiv:2412.08905. ↗
  2. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4232–4267, Singapore. Association for Computational Linguistics.
  3. Farid Ariai, Joel Mackenzie, and Gianluca Demartini. 2025. Natural language processing for the legal domain: A survey of tasks, datasets, models, and challenges. ACM Comput. Surv., 58(6). find
  4. Claire Barale, Mark Klaisoongnoen, Pasquale Minervini, Michael Rovatsos, and Nehal Bhuta. 2023. AsyLex: A dataset for legal language processing of refugee claims. In Proceedings of the Natural Legal Language Processing Workshop 2023, pages 244–257, Singapore. Association for Computational Linguistics. find
  5. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. 2025. LLMs instead of human judges? a large scale empirical study across 20 NLP evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 238–255, Vienna, Austria. Association for Computational Linguistics.
  6. Frederik Bay-Jørgensen, Henrik P Olsen, Mohammad N Jahromi, B. M Thomas, and Thomas Gammeltoft-Hansen. 2026. Managing fuzziness: Leveraging llms for discovering credibility indicators in asylum cases. Preprint, SocArXiv:r29fv_v1. find
  7. Michala Clante Bendixen. 2020. WELL-FOUNDED FEAR – CREDIBILITY AND RISK ASSESSMENT IN DANISH ASYLUM CASES. Refugees Welcome. find
  8. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, and 26 others. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261. ↗
  9. Thomas Davidson. 2024. Start generating: Harnessing generative artificial intelligence for sociological research. Socius, 10:23780231241259651. find
  10. Ricardo Dominguez-Olmedo, Vedant Nanda, Rediet Abebe, Stefan Bechtold, Christoph Engel, Jens Frankenreiter, Krishna P. Gummadi, Moritz Hardt, and Michael Livermore. 2025. Lawma: The power of specialization for legal annotation. In The Thirteenth International Conference on Learning Representations. find
  11. Jakub Drápal, Hannes Westermann, and Jaromir Savelka. 2023. Using large language models to support thematic analysis in empirical legal studies. Preprint, arXiv:2310.18729. ↗
  12. Igor Dugac and Tilmann Altwicker. 2025. Classifying legal interpretations using large language models. Artificial Intelligence and Law. find
  13. Ariel Ekgren, Amaru Cuba Gyllensten, Felix Stollenwerk, Joey Öhman, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, Judit Casademont, and Magnus Sahlgren. 2024. GPT-SW3: An autoregressive language model for the Scandinavian languages. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7886–7900, Torino, Italia. ELRA and ICCL. find
  14. Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  15. Andrew Halterman and Katherine A. Keith. 2025. Codebook llms: Evaluating llms as measurement tools for political science concepts. Political Analysis, page 1–17. find
  16. Lena Held and Ivan Habernal. 2025. Contemporary LLMs struggle with extracting formal legal arguments. In Proceedings of the Natural Legal Language Processing Workshop 2025, pages 292–303, Suzhou, China. Association for Computational Linguistics. find
  17. Maya Ellen Hertz and Asta Sofie Stage Jarlner. 2025. Trans “enough” for protection? experimenting with credibility in refugee status determination. Frontiers in Human Dynamics, Volume 7 - 2025. find
  18. Anna Høgenhaug, Thomas Gammeltoft-Hansen, and Asta Sofie Stage Jarlner. 2023. Nordic asylum practice in relation to religious conversion: Insights from denmark, norway and sweden. Legal and Protection Policy Research Series. find
  19. Andong Hua, Kenan Tang, Chenhe Gu, Jindong Gu, Eric Wong, and Yao Qin. 2025. Flaw or artifact? rethinking prompt sensitivity in evaluating LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19889–19899, Suzhou, China. Association for Computational Linguistics. find
  20. IBM. 2025. Ibm granite 4.0: hyper-efficient, high performance hybrid models for enterprise. find
  21. Asta S Stage Jarlner, Maya Ellen Hertz, Marieke Anne Heyl, Thomas Gammeltoft-Hansen, and William Hamilton Byrne. 2026. Credibility as a fuzzy concept in refugee law: a systematic literature review. Journal of Ethnic and Migration Studies, pages 1–31. find
  22. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.
  23. Kenneth Enevoldsen, Lasse Hansen, Dan S. Nielsen, Rasmus A. F. Egebæk, Søren V. Holm, Martin C. Nielsen, Martin Bernstorff, Rasmus Larsen, Peter B. Jørgensen, Malte Højmark-Bertelsen, Peter B. Vahlstrup, Per Møldrup-Dalum, and Kristoffer Nielbo. 2023. Danish foundation models. Preprint, arXiv:2311.07264. ↗
  24. Viet Dac Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13171–13189, Singapore. Association for Computational Linguistics. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, Alexandre Sablayrolles, Amélie Héliou, Amos You, Andy Ehrenberg, Andy Lo, Anton Eliseev, Antonia Calvi, Avinash Sooriyarachchi, Baptiste Bout, and 101 others. 2026. Ministral 3. Preprint, arXiv:2601.08584. Advances in Natural Language Processing - Natural Language Processing in the Generative AI Era, pages 1074–1084, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. ↗
  25. Laura Majer and Jan Šnajder. 2024. Claim check-worthiness detection: How well do LLMs grasp annotation guidelines? In Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), pages 245–263, Miami, Florida, USA. Association for Computational Linguistics. find
  26. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, and 4 others. 2025. Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, Vienna, Austria. Association for Computational Linguistics.
  27. Taylor Meizlish and Chris Ziffo. 2026. Evaluating an llm’s performance in annotating discourse strategies. Corpus Pragmatics, 10(1):23. find
  28. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933–949. find
  29. Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Adrian Gwoździej, Krzysztof Wróbel, SpeakLeash Team, and Cyfronet Team. 2025a. Bielik-11b-v3.0-instruct model card. find
  30. Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Krzysztof Wróbel, and Adrian Gwoździej. 2025b. Bielik 11b v3: Multilingual large language model for european languages. Preprint, arXiv:2601.11579. ↗
  31. Maja Pavlovic and Massimo Poesio. 2024. The effectiveness of LLMs as annotators: A comparative overview and empirical analysis of direct representation. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024, pages 100–110, Torino, Italia. ELRA and ICCL.
  32. Miguel Moura Ramos, Duarte M. Alves, Hippolyte Gisserot-Boukhlef, João Alves, Pedro Henrique Martins, Patrick Fernandes, José Pombal, Nuno M. Guerreiro, Ricardo Rei, Nicolas Boizard, Amin Farajian, Mateusz Klimaszewski, José G. C. de Souza, Barry Haddow, François Yvon, Pierre Colombo, Alexandra Birch, and André F. T. Martins. 2026. Eurollm-22b: Technical report. Preprint, arXiv:2602.05879. ↗
  33. Trine Rask Nielsen and Naja Holten Møller. 2022. Data as a lens for understanding what constitutes credibility in asylum decision-making. Proc. ACM Hum.-Comput. Interact., 6(GROUP). find
  34. Mattes Ruckdeschel. 2025. Just read the codebook! make use of quality codebooks in zero-shot classification of multilabel frame datasets. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6317–6337, Abu Dhabi, UAE. Association for Computational Linguistics. find
  35. Maximilian Schall and Gerard de Melo. 2025. The hidden cost of structure: How constrained decoding affects language model performance. In Proceedings of the 15th International Conference on Recent Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786. ↗
  36. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  37. Petter Törnberg. 2024. Best practices for text annotation with large language models. Preprint, arXiv:2402.05129. ↗
  38. Shubham Vatsal and Harsh Dubey. 2024. A survey of prompt engineering methods in large language models for different nlp tasks. Preprint, arXiv:2407.12994. ↗
  39. Yuqing Wang and Yun Zhao. 2024. Metacognitive prompting improves understanding in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1914–1926, Mexico City, Mexico. Association for Computational Linguistics. find
  40. Chuanchi Wen, Paul Clough, Rachel Paton, and Rebecca Middleton. 2025. Leveraging large language models for thematic analysis: a case study in the charity sector. AI & SOCIETY, pages 1–18. find
  41. Brandon T. Willard and Rémi Louf. 2023. Efficient guided generation for large language models. Preprint, arXiv:2307.09702. ↗
  42. Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, and 13 others. 2025. MMLU-ProX: A multilingual benchmark for advanced large language model evaluation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1513–1532, Suzhou, China. Association for Computational Linguistics.
  43. Mike Zhang, Max Müller-Eberstein, Elisa Bassignana, and Rob van der Goot. 2024. Snakmodel: Lessons learned from training an open danish large language model. Preprint, arXiv:2412.12956. ↗
  44. Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. 2024. ProSA: Assessing and understanding the prompt sensitivity of LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1950–1976, Miami, Florida, USA. Association for Computational Linguistics. find
  45. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can large language models transform computational social science? Computational Linguistics, 50(1):237–291.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/renaux-2026-llms-as-annotators-credibility-assessment/