Elamur Research⌕
☰
home / papers / 2026 / Evaluating the Impact of Source Diversity for RAG in Historical Research

Evaluating the Impact of Source Diversity for RAG in Historical Research

Summary

Historians increasingly use LLMs, and retrieval-augmented generation is often presented as a fix for hallucination. However, archival documents carry the viewpoints of the people who produced them, and RAG may pass those viewpoints on to the model's answers. This study asks whether the country and language of retrieved sources change how models portray a contested historical figure. Napoleon Bonaparte serves as the proof-of-concept case. The authors built a multilingual (English, French, Dutch) collection of newspapers, magazines, books and legal placards from 1805–1819, originating in the UK, US, France and the Netherlands. They filtered it for Napoleon-related material and posed ten historian-designed questions to three Qwen3 models of different sizes, with and without retrieval. Answers were evaluated with BERTScore and ROUGE-L, frame-semantic analysis, and syntactic profiling. Similarity scores stayed high throughout, yet frame analysis revealed clear shifts in perspective. Without retrieval, answers converged on a homogenised view across languages. With RAG, they diversified, and the pattern differed by language. The authors conclude that standard metrics overlook viewpoint effects and that retrieval actively reshapes, rather than neutrally supplements, model perspective.

Key points

  • The corpus draws on the British Library, Bibliothèque National de France, KB National Library of the Netherlands, the American Stories dataset, and Dutch-Indonesian placard books.
  • The initial collection held over 2.2 million paragraph chunks. BERTopic filtering kept only about 0.41% as Napoleon-relevant.
  • English chunks dominate, followed by Dutch and then French. This partly reflects Napoleonic-era censorship decrees in France (1800) and the Netherlands (1810), and the authors deliberately left the data unbalanced.
  • The evaluation combines BERTScore/ROUGE-L with frame semantics (partly following Minnema et al., 2022) and syntactic profiling.
  • The main finding is that high surface similarity masks substantial, language-specific perspective changes introduced by RAG.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4yz9z3uvzasd

@inproceedings{mahadeshwar2026evaluating,
  title = {{Evaluating the Impact of Source Diversity for RAG in Historical Research}},
  author = {Mahadeshwar, Ruhi and Cranenburgh, Andreas van and Caselli, Tommaso and Nissim, Malvina},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4yz9z3uvzasd},
  url = {https://doi.org/10.63317/4yz9z3uvzasd}
}

References

54 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sayantan Adak, Pauras Mangesh Meher, Paramita Das, and Animesh Mukherjee. 2025. REVerSum: A multi-staged retrieval-augmented generation method to enhance Wikipedia tail biographies through personal narratives. In Proceedings of the 31st International Conference on Computational Linguistics: Industry… find
  2. Utkarsh Agarwal, Kumar Tanmay, Aditi Khandelwal, and Monojit Choudhury. 2024. Ethical reasoning and moral value alignment of LLMs depend on the language we prompt them in. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-… find
  3. Mieke Bal. 2009. Narratology: Introduction to the theory of narrative. University of Toronto Press. find
  4. Peter Brett, Pascale Mompoint-Gaillard, Maria Helena Salema, and Sarah Keating-Chetwynd. 2009. How all teachers can support citizenship and human rights education: a framework for the development of competences. Council of Europe. find
  5. Dominique Brunato, Andrea Cimino, Felice Dell’Orletta, Giulia Venturi, and Simonetta Montemagni. 2020. Profiling-UD: a Tool for Linguistic Profiling of Texts. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7145–7151, Marseille, France. European Language Resources A… find
  6. Bojana Budimir. 2025. The challenge of translating culture-specific items: Evaluating MT and LLMs compared to human translators. In Proceedings of Machine Translation Summit XX: Volume 1, pages 455–467, Geneva, Switzerland. European Association for Machine Translation. find
  7. Yang Trista Cao, Anna Sotnikova, Jieyu Zhao, Linda X. Zou, Rachel Rudinger, and Hal Daumé III. 2025. Multilingual Large Language Models Leak Human Stereotypes across Language Boundaries. In Proceedings of the Fourth Workshop on NLP for Positive Impact (NLP4PI), pages 175–188, Vienna, Austria. Associ… find
  8. Fabio Celli and Valerio Basile. 2024. History repeats: Historical phase recognition from short texts. In Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024), pages 198–204, Pisa, Italy. CEUR Workshop Proceedings. find
  9. Fabio Celli and Astik Samal. 2025. Monitoring historical cultural hacking in large language models. In TRUST-AI: The European Workshop on Trustworthy AI. Organized as part of the European Conference of Artificial Intelligence - ECAI 2025., pages 1–10. find
  10. Ho-Chun Herbert Chang, Tracy Weener, Yung-Chun Chen, Sean Noh, Mingyue Zha, and Hsuan Lo. 2025. Do language models favor their home countries? Asymmetric propagation of positive misinformation and foreign influence audits. Harvard Kennedy School (HKS) Misinformation Review. find
  11. David Chanin. 2023. Open-source frame semantic parsing. find our card
  12. Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. 2024. Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. Journal of Legal Analysis, 16(1):64–93. find our card
  13. Gavin Daly. 2018. British soldiers and the legend of Napoleon. The Historical Journal, 61(1):131– 153. find
  14. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss library. find our card
  15. Philip Dwyer. 2015. Napoleon, The Revolution, and The Empire. In The Oxford Handbook of the French Revolution. Oxford University Press. find
  16. Philip G Dwyer. 2004. Napoleon Bonaparte as hero and saviour: image, rhetoric and behaviour in the construction of a legend. French History, 18(4):379–403. find
  17. Charles Fillmore. 1985. Frames and the semantics of understanding. Quaderni di semantica, 6:222– 254. find
  18. Charles J Fillmore et al. 2006. Frame semantics. Cognitive linguistics: Basic readings, 34:373– 400. find
  19. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097–1179. find our card
  20. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv e-prints, pages arXiv–2312. find our card
  21. Giselle Gonzalez Garcia and Christian Weilbach. 2023. If the sources could talk: Evaluating large language models for research assistance in history. In Computational Humanities Research Conference. find
  22. Tsafrir Goldberg and Yiftach Ron. 2014. ‘Look, Each Side Says Something Different’: the impact of competing history teaching approaches on Jewish and Arab adolescents’ discussions of the Jewish–Arab conflict. Journal of Peace Education, 11(1):1–29. find
  23. Tsafrir Goldberg and Geerte M Savenije. 2018. Teaching controversial historical issues. The Wiley International Handbook of History Teaching and Learning, pages 503–526. find
  24. Carlos-Emiliano González-Gallardo, Emanuela Boros, Nancy Girdhar, Ahmed Hamdi, Jose G. Moreno, and Antoine Doucet. 2024. Yes but.. Can ChatGPT Identify Entities in Historical Documents? In Proceedings of the 2023 ACM/IEEE Joint Conference on Digital Libraries, JCDL ’23, page 184–189. IEEE Press. find
  25. Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. find our card
  26. Marcus Hartner. 2014. Multiperspectivity. In Peter Huḧn et al., editor, The living handbook of narratology. Hamburg University. find
  27. Jakob Hauser, Daniel Kondor, Jenny Reddish, Majid Benam, Enrico Cioni, Federica Villa, James Bennett, Daniel Hoyer, Pieter Francois, Peter Turchin, et al. 2024. Large language models’ expert-level global history knowledge benchmark (HiST-LLM). Advances in Neural Information Processing Systems, 37:32… find
  28. Piotr Jaskulski, Tomasz Latos, Mariusz Ryńca, and Adam Zapała. 2025. Reliability of large language models as a tool for knowledge extraction from biographical dictionaries: the case of the Polish Biographical Dictionary. Digital Scholarship in the Humanities, 40(2):538–548. find
  29. Lotte Jensen. 2011. The Dutch against Napoleon. Resistance Literature and National Identity, 1806-1813. Journal of Dutch Literature, 2(2). find
  30. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. ACM Comput. Surv., 55(12). find our card
  31. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547. find our card
  32. Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In Proceedings of The ACM Collective Intelligence Conference, CI ’23, page 12–24, New York, NY, USA. Association for Computing Machinery. find
  33. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Proc… find
  34. Nianqi Li, Siyu Yuan, Jiangjie Chen, Jiaqing Liang, Feng Wei, Zujie Liang, Deqing Yang, and Yanghua Xiao. 2025. Past meets present: Creating historical analogy with large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper… find
  35. Alan McCully. 2012. History teaching, conflict and the legacy of the past. Education, Citizenship and Social Justice, 7(2):145–159. find
  36. Gosse Minnema, Gaetana Ruggiero, Marion Bartl, Sara Gemelli, Tommaso Caselli, Chiara Zanchi, Viviana Patti, Marco te Brömmelstroet, and Malvina Nissim. 2022. Responsibility framing under the magnifying lens of NLP: The case of gender-based violence and traffic danger. Computational Linguistics in th… find
  37. Margaret Mitchell, Giuseppe Attanasio, Ioana Baldini, Miruna Clinciu, Jordan Clive, Pieter Delobelle, Manan Dey, Sil Hamilton, Timm Dill, Jad Doughman, Ritam Dutt, Avijit Ghosh, Jessica Zosa Forde, Carolin Holtermann, Lucie-Aimée Kaffee, Tanmay Laud, Anne Lauscher, Roberto L Lopez-Davila, Maraim Mas… find
  38. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  39. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang… find our card
  40. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784–3803, Punta Cana, Dominican Republic. Association for Computational Linguisti… find our card
  41. Robert A Stradling. 2003. Multiperspectivity in history teaching: A guide for teachers. Council of Europe Strasbourg. find
  42. Marco Antonio Stranisci, Rossana Damiano, Enrico Mensa, Viviana Patti, Daniele Radicioni, and Tommaso Caselli. 2023. WikiBio: a semantic resource for the intersectional analysis of biographical events. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume… find
  43. Yi Sun, Wanru Yang, and Yin Liu. 2024. The Application of Constructing Knowledge Graph of Oral Historical Archives Resources Based on LLM-RAG. In Proceedings of the 2024 8th International Conference on Information System and Data Mining, ICISDM ’24, page 142–149, New York, NY, USA. Association for C… find
  44. The Trung Tran, Carlos-Emiliano González-Gallardo, and Antoine Doucet. 2025. Retrieval augmented generation for historical newspapers. find
  45. In Proceedings of the 24th ACM/IEEE Joint Conference on Digital Libraries, New York, NY, USA. Association for Computing Machinery. find
  46. Eva Vanmassenhove, Dimitar Shterionov, and Andy Way. 2019. Lost in translation: Loss and decay of linguistic richness in machine translation. In Proceedings of Machine Translation Summit XVII: Research Track, pages 222–232, Dublin, Ireland. European Association for Machine Translation. find our card
  47. Yifan Zeng. 2024. HistoLens: An LLM-Powered Framework for Multi-Layered Analysis of Historical Texts–A Case Application of Yantie Lun. arXiv preprint arXiv:2411.09978. — [arXiv:2411.09978](https://arxiv.org/abs/2411.09978) find
  48. Jing Zhou, Li Si, and Wenjun Hou. 2025. Humanities-in-the-loop: Using close reading as a method for retrieval-augmented generation (rag). Proceedings of the Association for Information Science and Technology, 62(1):1185–1189. 9. find
  49. Language Resource References BL. 2021. Digitised Books. c. 1510 - c. 1900. JSONL (OCR derived text + metadata). British Library Labs. find
  50. BnF. 2024. Documents de presse en mode texte du projet Europeana Newspapers (XIXe-XXe siècles). National Library of France. find
  51. Melissa Dell and Jacob Carlson and Tom Bryan and Emily Silcock and Abhishek Arora and Zejiang Shen and Luca D’Amico-Wong and Quan Le and Pablo Querubin and Leander Heldring. 2023. American Stories: A Large-Scale Structured Text Dataset of Historical U.S. Newspapers. arXiv. find
  52. KB. 2019. Delpher open krantenarchief (1.0). Koninklijke Bibliotheek. find
  53. KB. 2025. Digitale Bibliotheek voor de Nederlandse Letteren. Koninklijke Bibliotheek. find
  54. Romein, C.A. 2023. Nederlandsch-Indisch plakaat-boek, 1602-1811 (deel 1 t/m 17). Zenodo. 10. Appendix find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mahadeshwar-2026-evaluating-impact-source-diversity-rag/