Elamur Research⌕
☰
home / papers / 2019 / An annotated dataset of literary entities

An annotated dataset of literary entities

Paper · 2019 · David Bamman, Sejal Popat and Sheng Shen

Summary

Computational literary studies usually rely on NER tools trained on news corpora, because no labeled fiction data existed. This left their accuracy on novels unknown. The authors fill this gap with a new resource: roughly the opening 2,000 words of 100 public-domain English novels and short stories from Project Gutenberg (210,532 tokens), labeled with six ACE 2005 entity categories. The scheme covers common noun phrases as well as names and allows entities to sit inside other entities. It also adapts the ACE rules to metaphor, personified animals and metonymy. Compared with news, fiction devotes much more attention to people and physical settings. Using Ju et al.'s (2018) BiLSTM-CRF in its layered form, a model trained on ACE reaches only 45.7 F on literary test data, while training on the literary data gives 68.3. The ACE-trained model also misses female characters far more often than male ones, and this gap disappears with in-domain training. The resource therefore supports both literary NLP and cross-domain testing of NER robustness.

Key points

  • The corpus has 13,912 entity mentions across PER (9,383), FAC (2,154), LOC (1,170), GPE (878), VEH (197) and ORG (130). Pronouns are excluded, and the WEA category was dropped as too rare.
  • About 13.8% of annotations involve at least one level of nesting. Two annotators reached 86.0 F agreement on five texts that both of them labeled.
  • The books were split 80/10/10 by document. ACE→ACE scores 68.8 F, ACE→Literature 45.7 F, and Literature→Literature 68.3 F. PER and FAC each gain about 20 points with in-domain training.
  • Recall for women versus men is 38.0 vs 49.6 with ACE training (p = 0.0009) and 69.3 vs 68.2 with literary training. The ACE gap remains even after removing entities that begin with Mrs., Miss and Mr.
  • The data is released under CC BY-SA 4.0 at github.com/dbamman/litbank. Code is available at github.com/dbamman/NAACL2019-literary-entities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Literature and Literary Theory · 340 works

Cite

Cite as David Bamman, Sejal Popat and Sheng Shen. 2019. An annotated dataset of literary entities. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1220

@inproceedings{bamman2019annotated,
  title = {{An annotated dataset of literary entities}},
  author = {David Bamman, Sejal Popat and Sheng Shen},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1220},
  url = {https://doi.org/10.18653/v1/n19-1220}
}

References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Isabelle Augenstein, Leon Derczynski, and Kalina Bontcheva. 2017. Generalisation in named entity recognition. Comput. Speech Lang., 44(C):61–83. find
  2. David Bamman, Ted Underwood, and Noah A. Smith. 2014. A Bayesian mixed effects model of literary character. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 370–379, Baltimore, Maryland. Association for Computational Linguistic… our card
  3. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pages 4356–4364, USA. Curran… our card
  4. Mohit Iyyer, Anupam Guha, Snigdha Chaturvedi, Jordan Boyd-Graber, and Hal Daumé III. 2016. Feuding families and former friends: Unsupervised learning for dynamic fictional relationships. In North American Association for Computational Linguistics. find
  5. Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018. A neural layered model for nested named entity recognition. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 1… our card
  6. Svetlana Kiritchenko and Saif Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43–53. Association for Computational Linguistics. our card
  7. Julian Brooke, Adam Hammond, and Timothy Baldwin. 2016. Bootstrapped text-level named entity recognition for literature. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 344–350. Association for Computational Linguistics. find
  8. LDC. 2005. Ace (automatic content extraction) English annotation guidelines for entities. https://www.ldc.upenn.edu/ sites/www.ldc.upenn.edu/files/ english-entities-guidelines-v5.6. 6.pdf. ↗
  9. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  10. Wei Lu and Dan Roth. 2015. Joint mention extraction and classification with mention hypergraphs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 857–867. our card
  11. Snigdha Chaturvedi, Mohit Iyyer, and Hal Daumé III. 2017. Unsupervised learning of evolving relationships between literary characters. In Association for the Advancement of Artificial Intelligence. find
  12. Grant DeLozier, Ben Wing, Jason Baldridge, and Scott Nesbit. 2016. Creating a novel geolocation corpus from historical texts. In Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016), pages 188–198. Association for Computational Linguistics. find
  13. Elizabeth F. Evans and Matthew Wilkens. 2018. Nation, ethnicity, and the geography of British fiction, 1880–1940. Cultural Analytics. find
  14. Jenny Rose Finkel and Christopher D Manning. 2009. Nested named entity recognition. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, pages 141–150. Association for Computational Linguistics. our card
  15. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: the 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, NAACL-Short ’06, pages 57–60, Stroudsburg, PA, USA. Association for Computatio… find
  16. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. our card
  17. Aldrian Obaja Muis and Wei Lu. 2017. Labeling gaps between words: Recognizing overlapping mentions with mention separators. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2608–2618. Association for Computational Linguistics. find
  18. Paul Rayson, Alex Reinhold, James Butler, Chris Donaldson, Ian Gregory, and Joanna Taylor. 2017. A deeply annotated testbed for geographical text analysis: The corpus of lake district writing. In Proceedings of the 1st ACM SIGSPATIAL Workshop on Geospatial Humanities, pages 9–15. ACM. find
  19. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–1… our card
  20. Anders Søgaard. 2013. Estimating effect size across datasets. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 607–611. Association for Computational Linguistics. find
  21. Rachael Tatman. 2017. Gender and dialect bias in YouTube’s automatic captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 53–59, Valencia, Spain. Association for Computational Linguistics. our card
  22. Dennis Yi Tenen. 2018. Toward a computational archaeology of fictional space. New Literary History. find
  23. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of CoNLL-2003, pages 142–147. Edmonton, Canada. our card
  24. Ted Underwood, David Bamman, and Sabrina Lee. 2018. The transformation of gender in English-language fiction. Cultural Analytics. find
  25. Hardik Vala, David Jurgens, Andrew Piper, and Derek Ruths. 2015. Mr. Bennet, his coachman, and the Archbishop walk into a bar but only one of them gets recognized: On the difficulty of detecting characters in literary texts. In Proceedings of the 2015 Conference on Empirical Methods in Natural Langu… find
  26. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. find
  27. ACE 2005 multilingual training corpus. LDC. find
  28. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In NAACL. our card
  29. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bamman-2019-annotated-dataset-literary-entities/