Elamur Research⌕
☰
home / papers / 2024 / CoNLL#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 English

CoNLL#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 English

Paper · 2024 · Andrew Rueda, Elena Alvarez-Mellado and Constantine Lignos

Summary

NER scores on the CoNLL-03 English benchmark have stalled since about 2021. The authors ask whether the remaining errors reflect real model weaknesses or flaws in the benchmark itself. They label each of the 231 test documents with a domain (Sports, Economy, World Events) and a format (text article, data report, hybrid). Then they run three strong NER systems, XLM-R FLERT, LUKE and ASP-T0-3B, and break their F1 down by these document types. The breakdown shows that Economy, not the much-discussed sports material, is the weakest domain, especially for tabular data reports. Many apparent model errors turned out to be gold-label mistakes. So the authors gathered earlier correction efforts, adjudicated between them, and also fixed systematic processing bugs such as broken sentence boundaries and tokenization. They kept the original annotation guidelines and left the training and dev sets untouched. The result is CoNLL#, a corrected test set on which every model scores higher. It supports cleaner and more interpretable error analysis than the original data.

Key points

  • Test set composition: 101 Sports, 67 Economy and 63 World Events documents. By format, there are 139 text articles, 73 data reports and 19 hybrids. All World Events documents are plain text.
  • Replicated F1 on the original test set is roughly 93.6 for XLM-R FLERT, 94.4 for LUKE and 93.9 for ASP-T0-3B. All three models were trained on the uncorrected training set.
  • Economy data reports are the weakest slice: about 74.5 F1 for XLM-R FLERT, 75.4 for LUKE and 85.0 for ASP-T0-3B. Sports text and data scores mostly sit around 94–95.
  • Earlier corrections differ in scope and in what they fix:
    • CoNLL++ fixed 309 labels but left sentence boundaries and tokenization alone.
    • ReCoNLL fixed 105 labels and 10 sentence boundaries.
    • CoNLL-CODAIT changed 1,320 token labels across the whole corpus and in effect altered the guidelines.
    • CLEANCONLL, built from CoNLL-CODAIT, also modifies the train and dev sets.
  • CoNLL# is released at github.com/bltlab/conll-sharp. The authors' modifications are licensed CC BY 4.0, and a single trained linguist annotated and adjudicated the data.

Interesting ideas

  • I1. Tagging each test document with its genre and layout reveals where models actually fail, which a single aggregate F1 hides; here it exposed tabular economy text as the weak spot.
  • I2. Before blaming the model, check whether its "errors" are actually gold-standard mistakes, and fix the benchmark first so that error analysis has low noise.
  • I3. Corrected versions of one dataset often disagree, so the authors adjudicate among them and publish disagreement counts instead of picking one version blindly.
  • I4. Sentence-boundary and tokenization bugs are treated as first-class annotation errors, because they quietly corrupt label-level evaluation.
  • I5. Corrections follow the original guidelines and leave train and dev untouched, so new scores stay comparable with older results.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Tag our evaluation texts by genre and format (narrative, dialogue, lists, word problems with tables) and report per-slice scores, so regressions in rare formats stay visible.
  • I2 (tree): When gparse and the perceptron parser disagree with the treebank, add an adjudication pass that flags likely gold errors before counting them as parser failures.
  • I3 (learn): When several witness parsers or LLM teachers produce competing silver labels, record where they conflict and adjudicate explicitly, so the disagreement itself becomes a signal.
  • I4 (tokens): Audit our tokenizer and sentence splitter on tabular or score-like lines (e.g., "Chelsea 2 Everton 2"), where segmentation errors spread into tagging and parsing.
  • I5 (tree): When converting or repairing UD dialects in our treebanks, fix test data under the original conventions and version the changes separately, so parser scores remain comparable over time.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Andrew Rueda, Elena Alvarez-Mellado and Constantine Lignos. 2024. CoNLL#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 English. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{rueda2024conll,
  title = {{CoNLL\#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 English}},
  author = {Andrew Rueda, Elena Alvarez-Mellado and Constantine Lignos},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/2sh49fv6k8t3},
  url = {https://doi.org/10.63317/2sh49fv6k8t3}
}

References

18 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rodrigo Agerri and German Rigau. 2016. Robust multilingual named entity recognition with shallow semi-supervised features. Artif. Intell., 238:63–82. ↗
  2. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. ↗
  3. Nancy Chinchor and Beth Sundheim. 1993. MUC-5 evaluation metrics. In Fifth Message Understanding Conference (MUC-5): Proceedings of a Conference Held in Baltimore, Maryland, August 25-27, 1993. find
  4. Laura Chiticariu, Rajasekar Krishnamurthy, Yunyao Li, Frederick Reiss, and Shivakumar Vaithyanathan. 2010. Domain adaptation of rule-based annotators for named-entity recognition tasks. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1002–1012, Cambridge, MA. Association for Computational Linguistics. find
  5. Ajay Nagesh, Ganesh Ramakrishnan, Laura Chiticariu, Rajasekar Krishnamurthy, Ankush Dharkar, and Pushpak Bhattacharyya. 2012. Towards efficient named-entity rule induction for customizability. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 128–138, Jeju Island, Korea. Association for Computational Linguistics. find
  6. Chester Palen-Michel, Nolan Holley, and Constantine Lignos. 2021. SeqScore: Addressing barriers to reproducible named entity recognition evaluation. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 40–50, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  7. Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, 2(11). ↗
  8. Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. 2021. AI and the everything in the whole wide world benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran. ↗
  9. Susanna Rücker and Alan Akbik. 2023. Clean-CoNLL: A nearly noise-free named entity recognition dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8628–8645, Singapore. Association for Computational Linguistics. find
  10. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singapore. Association for Computational Linguistics.
  11. Stefan Schweter and Alan Akbik. 2020. FLERT: Document-level features for named entity recognition. arXiv preprint arXiv:2011.06993. ↗
  12. Tomasz Stanislawek, Anna Wróblewska, Alicja Wójcicka, Daniel Ziembicki, and Przemyslaw Biecek. 2019. Named entity recognition - is In Proceedings of there a glass ceiling? the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 624–633, Hong Kong, China. Association for Computational Linguistics. find
  13. Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6442–6454, Online. Association for Computational Linguistics. 11. find
  14. Language Resource References Jinlan Fu, Pengfei Liu, and Qi Zhang. 2020. Rethinking generalization of neural models: A named entity recognition case study. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7732–7739. find
  15. Tianyu Liu, Yuchen Eleanor Jiang, Nicholas Monath, Ryan Cotterell, and Mrinmaya Sachan. 2022. Autoregressive structured prediction with language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 993–1005, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  16. Frederick Reiss, Hong Xu, Bryan Cutler, Karthik Muthuraman, and Zachary Eichenberger. 2020. Identifying incorrect labels in the CoNLL-2003 corpus. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 215–226, Online. Association for Computational Linguistics. find
  17. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. ↗
  18. Zihan Wang, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, and Jiawei Han. 2019. Cross-Weigh: Training named entity tagger from imperfect annotations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5154–5163, Hong Kong, China. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rueda-2024-conll-fine-grained-error-analysis/