Elamur Research⌕
☰
home / papers / 2019 / NNE: A Dataset for Nested Named Entity Recognition in English Newswire

NNE: A Dataset for Nested Named Entity Recognition in English Newswire

Paper · 2019 · Nicky Ringland, Xiang Dai, Ben Hachey et al.

Summary

Most named entity recognizers produce only flat spans with coarse types. As a result, they miss what entities embedded inside other entities can tell us, such as a state inside a court's name or a job title inside a person mention. The authors address the shortage of large training data for nested NER by building NNE, a manually created annotation layer over the whole Wall Street Journal part of the Penn Treebank. They start from the flat BBN schema and grow its inventory from 64 to 114 types. Every structural sub-part of names, dates and numeric expressions is marked, using a custom tool that offers pre-annotation and propagates suggestions. The resource holds 279,795 mentions, nested up to six levels deep, and annotators agreed with each other at a high rate. In benchmarks, a hypergraph model reaches the best F1 but decodes slowly, while flat taggers lose much of their recall. Because NNE lines up with the PTB, it can be studied jointly with the syntactic and semantic layers that already exist for that corpus.

Key points

  • NNE contains 279,795 mentions of 114 entity types, with nesting depth up to 6. Of the 118,525 top-level mentions, 39.6% have no inner structure. A single span can carry several types: this is true of 19,144 of the 260,386 spans.
  • Four annotators spent 270 hours in total, plus 17 hours of adjudication. Sections 00 and 23 received double annotation, while every one of the four annotators labelled section 02.
  • Agreement was measured with Fleiss' kappa:
    • 0.907 over all tokens;
    • 0.832 over tokens inside at least one mention;
    • annotators scored an average P/R/F1 of 94.3/91.8/93.0 against the adjudicated gold standard.
  • The data split uses section 02 for development, sections 23–24 for test, and the remaining sections for training. Evaluation follows CoNLL-style exact matching.
  • Benchmark F1 scores:
    • Hypergraph (Wang and Lu, 2018): 91.4, at 9 words/s;
    • Transition-based model: 73.6, at 57 words/s;
    • BiLSTM-CRF-TOP: 53.5, with recall of only 38.0;
    • BiLSTM-CRF-BOTTOM: 74.7;
    • BiLSTM-CRF-BOTH: 88.9.

Interesting ideas

  • I1. Nested labels are collapsed into a per-token "tag stack" (e.g. STATE REGION), so ordinary token-level agreement statistics can be applied to tree-shaped annotation.
  • I2. When an entity has a genuinely dual nature, such as a hotel being both a business and a place, the scheme adds a dedicated category instead of forcing a choice between ORG and LOC.
  • I3. The same token always receives the same inner label, so 'Toronto' is marked CITY even inside a university's name. The nesting then exposes relations such as location and part-whole.
  • I4. Anything that looks like a person's name is labelled NAME without consulting outside knowledge, which keeps annotation decisions grounded in the surface text.
  • I5. Merging the outputs of an outermost-only flat tagger and an innermost-only flat tagger produces a surprisingly strong nested baseline at low cost.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): witness parsers' nested entity outputs could be compared via token stacks, giving a cheap agreement score for accepting or rejecting silver data.
  • I2 (world): the ontology could hold explicit dual-nature classes (hotel, building, media outlet) that inherit both organizational and locative properties, so frames don't pick one arbitrarily.
  • I3 (world): inner mentions could seed explainable facts in the text's world, such as "court located in Ontario" or "Southern California part of California".
  • I4 (frames): surface-evident labels could be kept separate from knowledge-backed claims, so every inference about whether a name really denotes a person is traceable.
  • I5 (tokens): two flat passes could be run in the tagger as a fast fallback before a slow nested decoder, with the hypergraph-style model reserved for disputed spans.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of SydneySydney · Australia45 works · 68,350 citations🔬Data61Sydney · Australia16 works · 2,532 citations🏛️Commonwealth Scientific and Industrial Research OrganisationCanberra · Australia15 works · 1,898 citations

Also on the byline: Digital Health Cooperative Research Centre.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksNamed entity recognition · 200 works

Cite

Cite as Nicky Ringland, Xiang Dai, Ben Hachey, Sarvnaz Karimi, Cecile Paris and James R. Curran. 2019. NNE: A Dataset for Nested Named Entity Recognition in English Newswire. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1510

@inproceedings{ringland2019nne,
  title = {{NNE: A Dataset for Nested Named Entity Recognition in English Newswire}},
  author = {Nicky Ringland, Xiang Dai, Ben Hachey, Sarvnaz Karimi, Cecile Paris and James R. Curran},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1510},
  url = {https://doi.org/10.18653/v1/p19-1510}
}

References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Beatrice Alex, Barry Haddow, and Claire Grover. 2007. Recognising nested named entities in biomedical text. In BioNLP, pages 65–72. find
  2. Kate Byrne. 2007. Nested named entity recognition in historical archive text. In ICSC, pages 589–596. find
  3. Jean Carletta. 1996. Assessing agreement on classification tasks: The Kappa statistic. Comput. Linguist., 22(2):249–254. ↗
  4. Xiang Dai. 2018. Recognizing complex entity mentions: A review and future directions. In ACL-SRW, pages 37–44. find
  5. Jessica Ficler and Yoav Goldberg. 2016. Coordination annotation extension in the Penn tree bank. In ACL, pages 834–842. find
  6. Jenny Rose Finkel and Christopher Manning. 2009. Nested named entity recognition. In EMNLP, pages 141–150.
  7. Heng Ji, Joel Nothman, and Ben Hachey. 2014. Overview of TAC-KBP2014 entity discovery and linking tasks. In TAC, pages 1333–1339. find
  8. Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018. A neural layered model for nested named entity recognition. In NAACL, pages 1446–1459.
  9. Arzoo Katiyar and Claire Cardie. 2018. Nested named entity recognition revisited. In NAACL, pages 861– 871.
  10. J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun ichi Tsujii. 2003. GENIA corpus a semantically annotated corpus for bio-textmining. Bioinformatics, 19:i180– i182. find
  11. Paul Kingsbury, Martha Palmer, and Mitch Marcus. 2002. Adding predicate argument structure to the Penn Treebank. In HLT, pages 252–256. find
  12. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In NAACL, pages 260–270. ↗
  13. Wei Lu and Dan Roth. 2015. Joint mention extraction and classification with mention hypergraphs. In EMNLP, pages 857–867.
  14. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Comput. Linguist., 19. ↗
  15. Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. The nombank project: An interim report.
  16. Alexis Mitchell, Stephanie Strassel, Shudong Huang, and Ramez Zakhary. 2004. ACE 2004 multilingual training corpus. LDC. ↗
  17. Aldrian Obaja Muis and Wei Lu. 2017. Labeling gaps between words: Recognizing overlapping mentions with mention separators. In EMNLP, pages 2608– 2618. find
  18. Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R Curran. 2013. Learning multilingual named entity recognition from Wikipedia. Artif. Intell., 194:151–175. ↗
  19. Nicky Ringland. 2016. Structured Named Entities. Ph.D. thesis, University of Sydney. find
  20. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In CoNLL. ↗
  21. Satoshi Sekine, Kiyoshi Sudo, and Chikashi Nobata. 2002. Extended named entity hierarchy. In LREC. find
  22. Mohammad Golam Sohrab and Makoto Miwa. 2018. Deep exhaustive model for nested named entity recognition. In EMNLP, pages 2843–2849.
  23. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. brat: a web-based tool for NLP-assisted text annotation. In EACL, pages 102–107.
  24. David Vadas and James Curran. 2007. Adding noun phrase structure to the Penn Treebank. In ACL, pages 240–247.
  25. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2005. ACE 2005 multilingual training corpus. LDC. ↗
  26. Bailin Wang and Wei Lu. 2018. Neural segmental hypergraphs for overlapping mention recognition. In EMNLP, pages 204–214. find
  27. Bailin Wang, Wei Lu, Yu Wang, and Hongxia Jin. 2018. A neural transition-based model for nested mention recognition. In EMNLP, pages 1011–1017. find
  28. Ralph Weischedel and Ada Brunstein. 2005. BBN pronoun coreference and entity type corpus. LDC. ↗
  29. Mingbin Xu, Hui Jiang, and Sedtawut Watcharawit-tayakul. 2017. A local detection approach for named entity recognition and mention detection. In ACL, pages 1237–1247. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ringland-2019-nne-dataset-nested-named-entity/