Elamur Research⌕
☰
home / papers / 2026 / Towards a Diagnostic and Predictive Evaluation Methodology for Sequence Labeling Tasks

Towards a Diagnostic and Predictive Evaluation Methodology for Sequence Labeling Tasks

Summary

Aggregate scores on in-distribution test sets show which system wins on average. They say little about why systems fail or how they will behave on data from another source. The authors address this gap for span identification tasks such as NER. Instead of scraping large amounts of real text, they build a compact, handcrafted test set. Its examples systematically vary span attributes such as shape, length, casing, quotation marks and sentence position, so scores can be broken down by attribute. They apply the approach to anglicism detection in Spanish and release BLAS (Benchmark for Loanwords and Anglicisms in Spanish), a test-only resource. They evaluate six systems: CRF, BETO, mBERT, two BiLSTM variants and a prompted Llama3. The per-attribute results reveal systematic weaknesses and help pick the right model for a given setting. Re-weighted to match the attribute mix of an external corpus, they also predict recall on COALAS and CALCS splits with a median correlation of 0.85. Small, linguistically designed benchmarks therefore offer an inexpensive way to diagnose models and forecast their performance.

Key points

  • The authors describe the methodology as diagnostic (it exposes weak phenomena), actionable (it guides model choice) and predictive (it forecasts out-of-distribution performance).
  • BLAS is deliberately unusual: it is small, built from scratch and designed to cover span variability exhaustively, unlike typical scraped benchmarks such as CoNLL03 or OntoNotes.
  • Raw BLAS recall is low for all models (CRF 6.50, BETO/mBERT/BiLSTMs about 23.5–23.8, Llama3 36.37). On its own, it correlates negatively with performance on external datasets (average about −0.15).
  • Extrapolated BLAS predictions correlate strongly with true recall on five external sets (COALAS test/dev, CALCS test A/B/dev): average 0.86, median 0.85. Model rankings correlate with an average of 0.75 and a median of 0.83.
  • Llama3 was prompted in Spanish with a task definition and examples, using a template inspired by Ashok and Lipton. It ranked last or near last on most external datasets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3f4yxedpym6a

@inproceedings{alvarezmellado2026diagnostic,
  title = {{Towards a Diagnostic and Predictive Evaluation Methodology for Sequence Labeling Tasks}},
  author = {Alvarez-Mellado, Elena and Gonzalo, Julio},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3f4yxedpym6a},
  url = {https://doi.org/10.63317/3f4yxedpym6a}
}

References

64 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Oshin Agarwal, Yinfei Yang, Byron C. Wallace, and Ani Nenkova. 2021. Entity-Switched Datasets: An Approach to Auditing the In-Domain Robustness of Named Entity Recognition Models. ArXiv:2004.04123 [cs]. — [arXiv:2004.04123](https://arxiv.org/abs/2004.04123) find
  2. AI@Meta. 2024. Llama 3 model card. find
  3. Beatrice Alex. 2008. Comparing corpus-based to web-based lookup techniques for automatic English inclusion detection. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC‘08), Marrakech, Morocco. European Language Resources Association (ELRA). find
  4. Elena Alvarez-Mellado. 2020. An annotated corpus of emerging anglicisms in Spanish newspaper headlines. In Proceedings of the 4th Workshop on Computational Approaches to Code Switching, pages 1–8, Marseille, France. European Language Resources Association. find
  5. Elena Álvarez-Mellado. 2020. Observatorio Lázaro - Observatorio del anglicismo en la prensa española. Online. find
  6. Elena Álvarez Mellado. 2025. Lexical borrowing detection as a sequence labeling task: Data, modeling and evaluation methods for anglicism retrieval in Spanish. Ph.D. thesis, Universidad Nacional de Educación a Distancia (UNED). find
  7. Elena Álvarez-Mellado and Julio Gonzalo. 2024. Characterizing spans for sequence labeling: A case on anglicism detection. Procesamiento del lenguaje natural, 73:235–246. https://adobo-task.github.io 5 https://github.com/lirondos/blas ↗
  8. Elena Álvarez-Mellado and Constantine Lignos. 2022. Detecting unassimilated borrowings in Spanish: An annotated corpus and approaches to modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3868–3888, Dublin, Ireland. Ass… find
  9. Elena Álvarez-Mellado, Jordi Porta-Zamorano, Constantine Lignos, and Julio Gonzalo. 2025. Overview of ADoBo at IberLEF 2025: Automatic detection of anglicisms in Spanish. Procesamiento del Lenguaje Natural, 75:373–383. find
  10. Gisle Andersen. 2012. Semi-automatic approaches to Anglicism detection in Norwegian corpus data. In Cristiano Furiassi, Virginia Pulcini, and Félix Rodríguez González, editors, The anglicization of European lexis, pages 111–130. John Benjamins. find
  11. Aymé Arango, Jorge Pérez, and Barbara Poblete. 2019. Hate speech detection is not as easy as you may think: A closer look at model validation. In Proceedings of the 42nd International ACM SI-GIR Conference on Research and Development in Information Retrieval, SIGIR’19, page 45–54, New York, NY, USA.… find
  12. Ron Artstein and Massimo Poesio. 2008. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555– 596. find
  13. Dhananjay Ashok and Zachary C. Lipton. 2023. PromptNER: Prompting For Named Entity Recognition. ArXiv:2305.15444 [cs]. find
  14. Gabriel Bernier-Colborne and Phillippe Langlais. 2020. HardEval: Focusing on challenging tokens to assess robustness of NER. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1704–1711, Marseille, France. European Language Resources Association. find
  15. Samuel R. Bowman and George Dahl. 2021. What will it take to fix benchmarking in natural language understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4843– 4855, Online. Association fo… find
  16. Luis Chiruzzo, Marvin Agüero-Torales, Gustavo Giménez-Lugo, Aldo Alvarez, Yliana Rodríguez, Santiago Góngora, and Thamar Solorio. 2023. find
  17. Overview of GUA-SPA at IberLEF 2023: Guarani-Spanish Code Switching Analysis. Procesamiento del Lenguaje Natural, 71:321–328. find
  18. Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M. Bender. 2017. Towards linguistically generalizable NLP systems: A workshop and shared task. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems, pages 1–10, Copenhagen, Denmark. Association for Computation… find
  19. Jinlan Fu, Pengfei Liu, and Graham Neubig. 2020. Interpretable multi-dataset evaluation for named entity recognition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6058– 6069, Online. Association for Computational Linguistics. find
  20. Cristiano Furiassi and Knut Hofland. 2007. The retrieval of false anglicisms in newspaper texts. In Corpus Linguistics 25 Years On, pages 347– 363. Brill Rodopi. find
  21. Abbas Ghaddar, Philippe Langlais, Ahmad Rashid, and Mehdi Rezagholizadeh. 2021. Context-aware adversarial training for name regularity bias in named entity recognition. Transactions of the Association for Computational Linguistics, 9:586–604. find
  22. Kyle Gorman and Steven Bedrick. 2019. We need to talk about standard splits. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2786–2791, Florence, Italy. Association for Computational Linguistics. find
  23. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao,… find
  24. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… find
  25. Georg Heigold, Stalin Varanasi, Günter Neumann, and Josef van Genabith. 2018. How robust are character-based word embeddings in tagging and MT against wrod scramlbing or randdm nouse? In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research… find
  26. Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, et al. 2023. A taxonomy and review of generalization research in nlp. Nature Machine Intelligence, 5(10):1161– 1174. find
  27. Hannah Kermes. 2004. Text analysis meets computational lexicography. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 854–860, Geneva, Switzerland. COLING. find
  28. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dyna… find
  29. Venelin Kovatchev and Matthew Lease. 2024. Benchmark transparency: Measuring the impact of data on evaluation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1536–1551, M… find
  30. Bill Yuchen Lin, Wenyang Gao, Jun Yan, Ryan Moreno, and Xiang Ren. 2021. RockNER: A simple method to create adversarial examples for evaluating the robustness of named entity recognition models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3728–373… find
  31. Hongyu Lin, Yaojie Lu, Jialong Tang, Xianpei Han, Le Sun, Zhicheng Wei, and Nicholas Jing Yuan. 2020. A rigorous study on named entity recognition: Can fine-tuning pretrained model lead to the promised land? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (E… find
  32. Tal Linzen. 2020. How can we accelerate progress towards human-like linguistic generalization? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5210–5217, Online. Association for Computational Linguistics. find our card
  33. Gyri Smordal Losnegaard and Gunn Inger Lyse. 2012. A data-driven approach to anglicism identification in Norwegian. In Gisle Andersen, editor, Exploring Newspaper Language: Using the web to create and investigate a large corpus of modern Norwegian, pages 131–154. John Benjamins Publishing. find
  34. Ruotian Ma, Xiaolei Wang, Xin Zhou, Qi Zhang, and Xuanjing Huang. 2023. Towards building more robust NER datasets: An empirical study on NER dataset bias from a dataset difficulty view. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4616–4630, Singap… find
  35. Stephen Mayhew, Tatiana Tsygankova, and Dan Roth. 2019. ner and pos when nothing is capitalized. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6256–6261, Hong K… find
  36. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computation… find
  37. Marcin Namysl, Sven Behnke, and Joachim Köhler. 2021. Empirical error modeling improves robustness of noisy neural sequence labeling. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 314–329, Online. Association for Computational Linguistics. find
  38. Sean Papay, Roman Klinger, and Sebastian Padó. 2020. Dissecting span identification tasks with performance prediction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4881– 4895, Online. Association for Computational Linguistics. find
  39. Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns, 2(11). Publisher: Elsevier. find
  40. Barbara Plank. 2016. What to do about non-standard (or non-canonical) language in NLP. In Proceedings of the 13th Conference on Natural Language Processing (KONVENS 2016), pages 13–20, Bochum, Germany. find
  41. Ehud Reiter. 2025. We should evaluate real-world impact. Computational Linguistics, pages 1–13. find
  42. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856–865, Melbourne, Australia. Association for C… find
  43. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912, Online. Association for Computational Linguistic… find
  44. Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, and Jordan Boyd-Graber. 2021. Evaluation examples are not equally informative: How should that change NLP leaderboards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the… find
  45. Félix Rodríguez González. 2002. Spanish. In Manfred Görlach, editor, English in Europe, chapter 7, pages 128–150. Oxford University Press. find
  46. F. Rodríguez González and A. Lillo Buades. 2009. Nuevo diccionario de anglicismos. Biblioteca románica hispánica: Diccionarios. Gredos. find
  47. Andrew Rueda, Elena Alvarez-Mellado, and Constantine Lignos. 2024. CoNLL\#: Fine-grained error analysis and a corrected test set for CoNLL-03 English. In Proceedings of the 2024 find
  48. Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3718–3728, Torino, Italia. ELRA and ICCL. find
  49. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singap… find our card
  50. Jacqueline Rae Larsen Serigos. 2017. Applying corpus and computational methods to loanword research: new approaches to Anglicisms in Spanish. Ph.D. thesis, The University of Texas at Austin. find
  51. Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. 2021. We need to talk about random splits. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1823–1832, Online. Association for Computational Lingu… find
  52. Tomasz Stanislawek, Anna Wróblewska, Alicja Wójcicka, Daniel Ziembicki, and Przemyslaw Biecek. 2019. Named entity recognition - is there a glass ceiling? In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 624–633, Hong Kong, China. Association for Computa… find
  53. Yulia Tsvetkov, Waleed Ammar, and Chris Dyer. 2015. Constraint-based models of lexical borrowing. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 598–608, Denver, Colorado. Association for Comput… find
  54. Jingxuan Tu and Constantine Lignos. 2021. TMR: Evaluating NER recall on tough mentions. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 155–163, Online. Association for Computational Linguistics. find
  55. Sowmya Vajjala and Ramya Balasubramaniam. 2022. What do we really know about state of the art NER? In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5983–5993, Marseille, France. European Language Resources Association. find
  56. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In Proceedings of the 33rd International Conference on Neural Information Processing Syst… find our card
  57. Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, Qinzhuo Wu, Zhengyan Li, Chong Zhang, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin, Bolin Zhu, Xiaoyu Xin… find
  58. Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2019. Errudite: Scalable, reproducible, and testable error analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 747–763, Florence, Italy. Association for Computational Linguistic… find
  59. Lexin Zhou, Pablo A. Moreno-Casares, Fernando Martínez-Plumed, John Burden, Ryan Burnell, Lucy Cheke, Cèsar Ferri, Alexandru Marcoci, Behzad Mehrbakhsh, Yael Moros-Daval, Seán Ó hÉigeartaigh, Danaja Rutar, Wout Schellaert, Konstantinos Voudouris, and José Hernández-Orallo. 2023. Predictable Artifici… — [arXiv:2310.06167](https://arxiv.org/abs/2310.06167) find
  60. Kiela, Douwe and Bartolo, Max and Nie, Yixin and Kaushik, Divyansh and Geiger, Atticus and Wu, Zhengxuan and Vidgen, Bertie and Prasad, Grusha and Singh, Amanpreet and Ringshia, Pratik and Ma, Zhiyi and Thrush, Tristan and Riedel, Sebastian and Waseem, Zeerak and Stenetorp, Pontus and Jia, Robin and… find
  61. Levesque, Hector J. and Davis, Ernest and Morgenstern, Leora. 2012. The Winograd schema challenge. AAAI Press, KR’12. find
  62. Pradhan, Sameer and Moschitti, Alessandro and Xue, Nianwen and Ng, Hwee Tou and Björkelund, Anders and Uryupina, Olga and Zhang, Yuchen and Zhong, Zhi. 2013. Towards Robust Linguistic Analysis using OntoNotes. Association for Computational Linguistics. find
  63. Rueda, Andrew and Alvarez-Mellado, Elena and Lignos, Constantine. 2024. CoNLL\#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 English. ELRA and ICCL. find
  64. Tjong Kim Sang, Erik F. and De Meulder, Fien. 2003. Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. find

↑ back to top

This page in Markdown →