Elamur Research⌕
☰
home / papers / 2026 / Evaluating the Adaptability of Large Language Models to Linguistic Variation

Evaluating the Adaptability of Large Language Models to Linguistic Variation

Summary

LLMs are often expected to handle any kind of text, but their ability to cope with shifts in genre has received little systematic study. The authors test this using French Named Entity Recognition (NER) on NEM.fr, a new gold-annotated corpus that greatly extends FENEC. It covers 11 text types, from legal, biomedical and encyclopedic writing to poetry, presidential speeches, emails and tweets. They evaluate one reasoning-oriented model, DeepSeek R1, under six prompting setups: zero-, one- and few-shot, each with and without chain-of-thought. The tag set, prompt format, post-processing and scoring stay fixed, so genre is the only factor that varies. Results are reported with strict and fuzzy F1. Once the model has learned the expected output format, the prompting strategy changes results very little. Genre, by contrast, produces large gaps: fuzzy F1 is around 0.85 on formal texts and drops below 0.20 on informal ones. The study argues that genre should be a central dimension when judging LLM robustness, and it releases a freely available multi-genre French evaluation resource.

Key points

  • NEM.fr corpus: 66 documents in 11 genres. The per-genre table totals 48,596 tokens, while the text elsewhere cites roughly 51,000. Its predecessor, FENEC, had about 15,000 tokens, 15 documents and 6 categories.
  • Annotation: done in INCEpTION following the Quæro guidelines, using six tags (PERS, ORG, LOC, EVENT, TIME, PROD), with AMOUNT dropped. Five annotators worked over about three months, with two annotating and a third curating at each stage.
  • Sources: new material includes texts from MORFITT, POPCORN, ESLO, TREMOLO and WIKINER-FR-GOLD, plus court decisions, presidential speeches and anonymized emails. Prose additions favour recent works by women authors to correct a male skew in FENEC.
  • Tag coverage: tweets are the only genre containing every one of the six entity types, whereas genres such as poetry and biomedical lack some types entirely.
  • Main result: prompting configuration matters little once the output format is learned, but genre strongly affects performance (fuzzy F1 from about 0.85 to under 0.20). Both the code and the corpus are published on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/57bpwacmcpr2

@inproceedings{xu2026evaluating,
  title = {{Evaluating the Adaptability of Large Language Models to Linguistic Variation}},
  author = {Xu, Ziyan and Seghier, Marina and Millour, Alice and Gonzalez-Gallardo, Carlos-Emiliano and Antoine, Jean-Yves},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/57bpwacmcpr2},
  url = {https://doi.org/10.63317/57bpwacmcpr2}
}

References

39 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anne Abeillé, Lionel Clément, and Alexandra Kinyon. 2000. Building a treebank for French. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00), Athens, Greece. European Language Resources Association (ELRA). find
  2. Lotfi Abouda and Olivier Baude. 2005. Du Français Fondamental aux ESLO. In Cahiers de linguistique, volume 33 of Cahiers de linguistique, pages 131–146, Lyon, France. find
  3. ATILF and CLLE. 2020. Corpus journalistique issu de l’est républicain. ORTOLANG (Open Resources and TOols for LANGuage) –www.ortolang.fr. find
  4. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72. find our card
  5. Daniel M. Bikel, Scott Miller, Richard Schwartz, and Ralph Weischedel. 1997. Nymble: a high-performance learning name-finder. In Fifth Conference on Applied Natural Language Processing, pages 194–201, Washington, DC, USA. Association for Computational Linguistics. find our card
  6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  7. Marie Candito, Benoît Crabbé, and Pascal Denis. 2010. Statistical French dependency parsing: treebank conversion and first results. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC 2010), pages 1840–1847, La Valletta, Malta. European Language Resource… find
  8. Marie Candito and Djamé Seddah. 2012a. Le corpus Sequoia : annotation syntaxique et exploitation pour l’adaptation d’analyseur par pont lexical. In TALN 2012 - 19e conférence sur le Traitement Automatique des Langues Naturelles, Grenoble, France. find
  9. Marie Candito and Djamé Seddah. 2012b. Le corpus Sequoia : annotation syntaxique et exploitation pour l’adaptation d’analyseur par pont lexical. In Actes de Traitement Automatique des Langues Naturelles (TALN), Grenoble, France. find
  10. Danrun Cao, Nicolas Béchet, and Pierre-François Marteau. 2024. WikiNER-fr-gold: A Gold-Standard NER Corpus. ArXiv:2411.00030 [cs]. find
  11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… find
  12. Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140–147, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. find
  14. Maud Ehrmann, Marco Turchi, and Ralf Steinberger. 2011. Building a multilingual named entity-annotated corpus using annotation projection. In Proceedings of the International Conference Recent Advances in Natural Language Processing 2011, pages 118–124. find
  15. Federico Errica, Giuseppe Siracusano, Davide Sanvito, and Roberto Bifulco. 2024. What did i do wrong? quantifying llms’ sensitivity and consistency to prompt engineering. arXiv preprint arXiv:2406.12334. find
  16. Olivier Galibert, Ludovic Quintard, Sophie Rosset, Pierre Zweigenbaum, Claire Nédellec, Sophie Aubin, Laurent Gillard, Jean Pierre Raysz, Delphine Pois, Xavier Tannier, Louise Deleger, and Dominique Laurent. 2010. Named and specific entity detection in varied data: the Quaero named entity baseline e… find
  17. Bastien Giordano, Maxime Prieur, Nakanyseth Vuth, Sylvain Verdy, Kévin Cousot, Gilles Sérasset, Guillaume Gadek, Didier Schwab, and Cédric Lopez. 2024. POPCORN: Fictional and Synthetic Intelligence Reports for Named Entity Recognition and Relation Extraction Tasks. Procedia Computer Science, 246:117… find
  18. G. Gravier, J-F. Bonastre, E. Geoffrois, S. Galliano, K. McTait, and K. Choukri. 2004. The ESTER evaluation campaign for the rich transcription of French broadcast news. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal. European L… find
  19. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. As… find
  20. Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. 2024. Does prompt formatting have any impact on llm performance? find
  21. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. find our card
  22. Yanis Labrak and Richard Dufour. 2022. ANTILLES: An Open French Linguistically Enriched Part-of-Speech Corpus. In 25th International Conference on Text, Speech and Dialogue (TSD), Brno, Czech Republic. find
  23. Yanis Labrak, Mickael Rouvier, and Richard Dufour. 2023. MORFITT : Un corpus multi-labels d’articles scientifiques français dans le domaine biomédical. find
  24. Anne Lacheret, Sylvain Kahane, Julie Beliao, Anne Dister, Kim Gerdes, Jean-Philippe Goldman, Nicolas Obin, Paola Pietrandrea, and Atanas Tchobanov. 2014. Rhapsodie: un Treebank annoté pour l’étude de l’interface syntaxe-prosodie en français parlé. In 4e Congrès Mondial de Linguistique Française, vol… find
  25. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. find
  26. Jingye Li, Hao Fei, Jiang Liu, Shengqiong Wu, Meishan Zhang, Chong Teng, Donghong Ji, and Fei Li. 2021. Unified named entity recognition as word-word relation classification. find
  27. Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2021. Crossner: Evaluating cross-domain named entity recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15):13452–13460. find
  28. Mónica Marrero, Julián Urbano, Sonia Sánchez-Cuadrado, Jorge Morato, and Juan Miguel Gómez-Berbís. 2013. Named entity recognition: fallacies, challenges and opportunities. Computer Standards & Interfaces, 35(5):482–489. find
  29. Jade Mekki, Delphine Battistelli, Gwénolé Lecorvé, and Nicolas Béchet. 2021. TREMoLo-Tweets corpus: guide d’annotation pour un corpus annoté en registres de langue pour le français. find
  30. Alice Millour, Yoann Dupont, Karen Fort, and Liam Duignan. 2024. Unveiling strengths and weaknesses of NLP systems based on a rich evaluation corpus: The case of NER in French. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (… find
  31. Alice Millour, Yoann Dupont, Alexane Jouglar, and Karën Fort. 2022. FENEC : un corpus à échantillons équilibrés pour l’évaluation des entités nommées en français. In Conférence sur le Traitement Automatique des Langues Naturelles (TALN), Avignon, France. find
  32. David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3–26. find
  33. Aurélie Névéol, Cyril Grouin, Jérémy Leixa, Sophie Rosset, and Pierre Zweigenbaum. 2014. The quaero french medical corpus : A ressource for medical entity recognition and normalization. find
  34. Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R. Curran. 2013. Learning multilingual named entity recognition from wikipedia. Artificial Intelligence, 194:151– 175. Artificial Intelligence, Wikipedia and Semi-Structured Resources. find
  35. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), System Demonstrations, pages 194–199. Association for C… find our card
  36. Sophie Rosset, Cyril Grouin, and Pierre Zweigenbaum. 2011. Entités nommées structurées: guide d’annotation Quaero. LIMSI-Centre national de la recherche scientifique. find
  37. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. find
  38. Djamé Seddah and Marie Candito. 2016. Hard time parsing questions: Building a QuestionBank for French. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2366–2370, Portorož, Slovenia. European Language Resources Association (ELRA). 9. find
  39. Language Resource References Seghier, Marina and Millour, Alice. 2026. NEM.fr: Named-Entities Multi-genre French Corpus. ISLRN Repository, ISLRN 857-654-609-197-8. PID https://www.islrn.org/resources/request/4126/. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-evaluating-adaptability-large-language-models/