Elamur Research⌕
☰
home / papers / 2022 / Comparing Annotated Datasets for Named Entity Recognition in English Literature

Comparing Annotated Datasets for Named Entity Recognition in English Literature

Paper · 2022 · Rositsa V. Ivanova, Sabrina Kirrane and Marieke van Erp

Summary

Named entity recognition benchmarks for English fiction follow different annotation conventions, so a tool's measured quality may reflect the chosen gold standard as much as the tool. The authors study this for English literature. They examine two public corpora: LitBank, which uses ACE 2005-style, multi-layered annotation of six entity types, and the OWTO dataset by Dekker et al., which covers persons only. They then build new gold standards for the 12 novel sections that appear in both corpora. Two annotators label these sections independently in Doccano under their own guidelines, which separate a PERSON label (based on MUC-7/CoNLL-2003 conventions) from a second label, PERX. The annotators then reconcile their versions. A domain-specific and a general-purpose NER tool are scored against all four gold standards, and the sources of disagreement are analysed. The main finding is that both tools' scores change substantially depending on the reference used. NER results in this domain are therefore only comparable when the annotation guidelines behind them are understood.

Key points

  • LitBank covers excerpts from 100 novels (210,532 tokens), drawn from Project Gutenberg. It has six ACE 2005 categories (PER, FAC, LOC, GPE, ORG, VEH) and nested layers with BIO prefixes.
  • OWTO has 20 modern and 20 old novels and annotates persons only. Two people annotated it, each working on different novels, with fragments of roughly 300 sentences. BookNLP output served as a pre-annotation to speed up the work.
  • The authors found that LitBank's publications disagree on how many annotators were involved. Checking the GitHub commit history did not resolve the question.
  • To score flat-output tools against LitBank, the nested layers were flattened by keeping the longest-span person entity. The authors acknowledge that this choice biases the resulting gold standard.
  • The new annotation of the 12 overlapping sections used Doccano v1.2.2 and two non-communicating annotators, whose differences were later adjudicated. Measured tool performance varied significantly across the four gold standards.

Interesting ideas

  • I1. Nested mentions such as "the cook" inside "the cook's sister" are each real entities, and collapsing them to one flat layer silently discards one; the authors make their collapse rule explicit and admit it shapes the reference.
  • I2. Scoring one tool against several gold standards that follow different guidelines exposes how much of an apparent error is just a convention mismatch.
  • I3. Annotating only the text that two corpora already share isolates the effect of guidelines, because the raw input is held constant.
  • I4. Pre-annotating with an existing tool speeds up labelling but can anchor annotators toward that tool's output, while independent double annotation followed by adjudication reduces this.
  • I5. The authors cross-checked a dataset's papers against its repository history and found undocumented inconsistencies in its provenance.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Keep nested noun-phrase entities as separate world objects linked by a possession relation, and treat any flat export as a separately documented view rather than the truth.
  • I2 (learn): Evaluate parsers and taggers against more than one reference treebank or dialect, and report the spread so that guideline effects are not mistaken for model regressions.
  • I3 (tree): When building UD dialect conversion rules, compare treebanks on overlapping sentences to learn systematic convention differences cleanly.
  • I4 (learn): Mark silver data seeded from one witness parser as potentially biased, and prefer agreement among independent witnesses plus a reconciliation step.
  • I5 (learn): Record annotator counts, guideline versions and commit hashes for every training or evaluation corpus, so that score changes can be traced to data changes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksNamed entity recognition · 200 works

Cite

Cite as Rositsa V. Ivanova, Sabrina Kirrane and Marieke van Erp. 2022. Comparing Annotated Datasets for Named Entity Recognition in English Literature. Proceedings of the Thirteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ivanova2022comparing,
  title = {{Comparing Annotated Datasets for Named Entity Recognition in English Literature}},
  author = {Rositsa V. Ivanova, Sabrina Kirrane and Marieke {van Erp}},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/2moyzgso4j5s},
  url = {https://doi.org/10.63317/2moyzgso4j5s}
}

References

34 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akbik, A., Bergmann, T., Blythe, D., Rasul, K., Schweter, S., and Vollgraf, R. (2019). Flair: An easy-to-use framework for state-of-the-art nlp. In NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54–59.
  2. Bamman, D., Underwood, T., and Smith, N. A. (2014). A bayesian mixed effects model of literary character. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 370–379.
  3. Bamman, D., Popat, S., and Shen, S. (2019). An annotated dataset of literary entities. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2138–2144. Association for Computational Linguistics, June.
  4. Bamman, D., Lewke, O., and Mansoor, A. (2020). An annotated dataset of coreference in English literature. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 44–54. European Language Resources Association, May. ↗
  5. BBN Technologies. (2007). Co-reference guidelines for english ontonotes version 7.0. https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/english-coreference-guidelines.pdf. Last accessed: 2021-04-18. ↗
  6. Chinchor, N., Brown, E., Ferro, L., and Robinson, P. (1999). Named entity recognition task definition. Mitre and SAIC. find
  7. Cordeiro, F. R. and Carneiro, G. (2020). A survey on deep learning with noisy labels: How to train your model when you cannot trust on the annotations? In 2020 33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pages 9–16. IEEE. find
  8. Dekker, N., Kuhn, T., and van Erp, M. (2019). Evaluating named entity recognition tools for extracting social networks from novels. PeerJ Computer Science, 5:e189. ↗
  9. Elson, D., Dames, N., and McKeown, K. (2010). Extracting social networks from literary fiction. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 138–147. ↗
  10. Finkel, J. R., Grenager, T., and Manning, C. (2005). Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 363–370. Association for Computational Linguistics.
  11. Frénay, B. and Verleysen, M. (2013). Classification in the presence of label noise: a survey. IEEE Transactions on Neural Networks and Learning Systems, 25(5):845–869. ↗
  12. Grishman, R. and Sundheim, B. M. (1996). Message understanding conference-6: A brief history.
  13. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics. find
  14. Hulpus, I., Hayes, C., Karnstedt, M., and Greene, D. (2013). Unsupervised graph-based topic labelling using dbpedia. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining, pages 465–474. find
  15. Krug, M., Reger, I., Jannidis, F., Weimer, L., Madarász, N., and Puppe, F. (2017). Overcoming data sparsity for relation detection in german novels. In DH. find
  16. Krug, M. (2020). Techniques for the Automatic Extraction of Character Networks in German Historic Novels. Doctoral thesis, Universität Würzburg. ↗
  17. Li, J., Sun, A., Han, J., and Li, C. (2020). A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering. ↗
  18. Linguistic Data Consortium. (2006). ACE (Automatic Content Extraction) English Annotation Guidelines for Entities. https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/english-entities-guidelines-v5.6.6.pdf. Last accessed: 2021-03-04. ↗
  19. Mac Kim, S. and Cassidy, S. (2015). Finding names in trove: named entity recognition for australian historical newspapers. In Proceedings of the Australasian Language Technology Association Workshop 2015, pages 57–65. find
  20. McHugh, M. L. (2012). Interrater reliability: The Kappa statistic. Biochemia medica, 22(3):276–282. ↗
  21. Nivre, J., Hall, J., Nilsson, J., Chanev, A., Eryigit, G., Kübler, S., Marinov, S., and Marsi, E. (2007). Maltparser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(2):95–135. ↗
  22. Poesio, M. and Artstein, R. (2005). The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 76–83.
  23. Pradhan, S., Ramshaw, L., Marcus, M., Palmer, M., Weischedel, R., and Xue, N. (2011). Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–27.
  24. Rösiger, I., Schulz, S., and Reiter, N. (2018). Towards coreference for literary text: Analyzing domain-specific phenomena. In Proceedings of the Second Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 129–138. Association for Computational Linguistics, August.
  25. Schweter, S. and Akbik, A. (2020). Flert: Document-level features for named entity recognition. arXiv preprint arXiv:2011.06993. ↗
  26. Sims, M., Park, J. H., and Bamman, D. (2019). Literary event detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3623–3634. Association for Computational Linguistics, July. find
  27. Tjong Kim Sang, E. F. and De Meulder, F. (2003). Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. ↗
  28. Toutanova, K., Klein, D., Manning, C. D., and Singer, Y. (2003). Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 173– 180. Association for Computational Linguistics.
  29. Vala, H., Jurgens, D., Piper, A., and Ruths, D. (2015). Mr. bennet, his coachman, and the archbishop walk into a bar but only one of them gets recognized: On the difficulty of detecting characters in literary texts. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 769–774. Association for Computational Linguistics, September. find
  30. van Cranenburgh, A. and Bod, R. (2017). A data-oriented model of literary language. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1228–1238. find
  31. Walker, C., Strassel, S., Medero, J., and Maeda, K. (2006). Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57:45. find
  32. Wang, Z., Shang, J., Liu, L., Lu, L., Liu, J., and Han, J. (2019). Crossweigh: Training named entity tagger from imperfect annotations. In Proc. 2019 Conferenc. on Empirical Methods in Natural Language Processing and the 9th International Joint Conf. on Natural Language Processing, EMNLP-IJCNLP 2019,, volume 1. ↗
  33. Woldenga-Racine, V. (2019). Issues in Named Entity Recognition on Early Modern English Letters. Master thesis, University of Washington. find
  34. Zhu, G. and Iglesias, C. A. (2018). Exploiting semantic similarity for named entity disambiguation in knowledge graphs. Expert Systems with Applications, 101:8–24. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ivanova-2022-comparing-annotated-datasets-named-entity/