Elamur Research⌕
☰
home / papers / 2022 / Building and Evaluating Universal Named-Entity Recognition English corpus

Building and Evaluating Universal Named-Entity Recognition English corpus

Paper · 2022 · Diego Alves, Gaurish Thakkar and Marko Tadic

Summary

Named-entity recognition does not have a shared cross-lingual annotation standard of the kind Universal Dependencies provides for POS tagging and syntax. Labels vary from corpus to corpus, and most corpora use only a few coarse types. The authors respond with the Universal Named Entity (UNER) framework, a deep four-level hierarchy derived from Sekine's taxonomy. They also build a fully automatic pipeline that produces UNER-labelled text from Wikipedia without any manual annotation.

The pipeline works in four steps:

  1. It extracts article text from Wikipedia dumps and keeps the hyperlinks.
  2. It looks up each linked entity's classes in DBpedia.
  3. It picks the most specific class using a priority ranking.
  4. It maps that class to a UNER tag in IOB format.

Applied to English, the pipeline labels about 85% of the inspected entities correctly. Against a manually annotated Golden set, precision is fairly high but recall is low, because many entity mentions in Wikipedia have no link. Several post-processing experiments were tested. The best one fills gaps with an entity dictionary whose DBpedia classes are checked against the Open Event Knowledge Graph (OEKG). The workflow is released openly and can be reused for any language that has both Wikipedia and DBpedia.

Key points

  • UNER v1 had 215 labels. The authors reduced it to UNER v2 (124 labels; 1/3/14/53/88 nodes from root to level 4), keeping only labels with a DBpedia counterpart. Most Time and Numerical expressions were dropped.
  • Labels chain the levels with hyphens, e.g. Name-Event-Natural Phenomenon-Earthquake, and are prefixed with B/I. Non-entity tokens get O.
  • Tools used: WikiExtractor, SPARQL queries through SPARQLWrapper, and a hand-built mapping from DBpedia classes to UNER tags. Code and data are published at github.com/cleopatra-itn/MIDAS.
  • The English corpus uses only 99 of the 124 labels. Only 62 of those have non-zero precision or recall against the Golden set.
  • Results on coarse categories (Person, Location, Organization, Event, …):
    • Baseline: P 76.9, R 25.1, F1 34.0.
    • OEKG-verified dictionary (experiment 4): P 74.6, R 27.3, F1 36.0.
    • Global dictionary: raises recall but makes precision collapse.

Interesting ideas

  • I1. When an entity carries several ontology classes, give each class a priority based on its depth in the ontology and keep the most specific one. This turns a messy class set into one deterministic label, and the rule is easy to inspect when it fails.
  • I2. Keep only sentences that contain at least one linked entity, so that unlinked text is not silently treated as non-entity. This reduces false negatives in automatically labelled data.
  • I3. Spread known entity strings from linked mentions to unlinked ones, but accept a match only if an external knowledge graph confirms the class. Unchecked spreading raises recall while destroying precision.
  • I4. Hyphen-joined hierarchical labels let one corpus be scored at both fine and coarse granularity. This shows whether errors come from picking the wrong branch or only the wrong leaf.
  • I5. Count how many labels in a taxonomy actually appear in the data and how many receive non-zero scores, then merge or prune the dead ones. This keeps the label set aligned with what the data can support.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Use depth-ranked selection to resolve one entity type out of the many WordNet/Wikidata classes it can receive, and log the losing classes so the choice remains explainable.
  • I2 (learn): When building silver data from partial annotations, drop the regions the witnesses never covered instead of treating them as negative examples.
  • I3 (learn): Run self-induced repair or propagation rules through an independent verifier (knowledge graph or second parser), and measure precision separately so that recall gains do not hide damage.
  • I4 (world): Store ontology types as full paths so that entity typing can be evaluated, and backed off, level by level.
  • I5 (frames): Audit VerbNet role and frame inventories against parsed corpora, and collapse roles that are never produced or never scored correctly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ZagrebZagreb · Croatia10 works · 1,179 citations🎓Faculty of Humanities and Social Sciences in ZagrebZagreb · Croatia4 works · 14 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksEvaluation & benchmarks · 1,220 worksNamed entity recognition · 200 works

Cite

Cite as Diego Alves, Gaurish Thakkar and Marko Tadic. 2022. Building and Evaluating Universal Named-Entity Recognition English corpus. arXiv:2212.07162.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2212.07162 · arXiv 2212.07162

@misc{alves2022building,
  title = {{Building and Evaluating Universal Named-Entity Recognition English corpus}},
  author = {Diego Alves, Gaurish Thakkar and Marko Tadic},
  year = {2022},
  doi = {10.48550/arxiv.2212.07162},
  eprint = {2212.07162},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2212.07162}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alves, D., Kuculo, T., Amaral, G., Thakkar, G., Tadić, M.: Uner: Universal named-entity recognitionframework. In: Proceedings of the 1st International Workshop on D. Alves et al. Cross-lingual Event-centric Open Analytics. pp. 72–79. Association for Computational Linguistics (2020), https://www.aclweb.org/anthology/W10-0712 ↗
  2. Alves, D., Thakkar, G., Tadić, M.: Evaluating language tools for fifteen EU-official under-resourced languages. In: Proceedings of the 12th Language Resources and Evaluation Conference. pp. 1866–1873. European Language Resources Association, Marseille, France (May 2020), https://www.aclweb.org/anthology/2020. lrec-1.230 ↗
  3. Bekavac, B., Tadić, M.: A generic method for multi word extraction from wikipedia. In: Proceedings of the 30th International Conference on Information Technology Interfaces (2008), https://www.bib.irb.hr/348724 ↗
  4. Eckart de Castilho, R., Mújdricza-Maydt, É., Yimam, S.M., Hartmann, S., Gurevych, I., Frank, A., Biemann, C.: A web-based tool for the integrated annotation of semantic and syntactic structures. In: Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH). pp. 76–84. The COLING 2016 Organizing Committee, Osaka, Japan (Dec 2016), https://www.aclweb.org/anthology/W16-4011 ↗
  5. Chinchor, N., Robinson, P.: Appendix E: MUC-7 Named Entity Task Definition (version 3.5). In: Seventh Message Understanding Conference (MUC-7): Proceedings of a Conference Held in Fairfax, Virginia, April 29 - May 1, 1998 (1998), https://www.aclweb.org/anthology/M98-1028 ↗
  6. Dumitrescu, S.D., Avram, A.: Introducing RONEC - the romanian named entity corpus. CoRR abs/1909.01247 (2019), http://arxiv.org/abs/1909.01247 ↗
  7. Freitas, C., Carvalho, P., Gonçalo Oliveira, H., Mota, C., Santos, D.: Second HAREM: advancing the state of the art of named entity recognition in Portuguese. In: quot; In Nicoletta Calzolari; Khalid Choukri; Bente Maegaard; Joseph Mariani; Jan Odijk; Stelios Piperidis; Mike Rosner; Daniel Tapias (ed) Proceedings of the International Conference on Language Resources and Evaluation (LREC 2010)(Valletta 17-23 May de 2010) European Language Resources Association. European Language Resources Association (2010) find
  8. Honnibal, M., Montani, I., Van Landeghem, S., Boyd, A.: spaCy: Industrial-strength Natural Language Processing in Python (2020). https://doi.org/10.5281/zenodo.1212303, https://doi.org/10.5281/zenodo.1212303 ↗
  9. Kim, S., Toutanova, K., Yu, H.: Multilingual named entity recognition using parallel data and metadata from Wikipedia. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 694–702. Association for Computational Linguistics, Jeju Island, Korea (Jul 2012), https://www.aclweb.org/anthology/P12-1073 ↗
  10. Lawson, N., Eustice, K., Perkowitz, M., Yetisgen-Yildiz, M.: Annotating large email datasets for named entity recognition with Mechanical Turk. In: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. pp. 71–79. Association for Computational Linguistics, Los Angeles (jun 2010), https://www.aclweb.org/anthology/W10-0712 ↗
  11. Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P.N., Hellmann, S., Morsey, M., van Kleef, P., Auer, S., Bizer, C.: Dbpedia - a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web 6(2), 167–195 (2015). https://doi.org/10.3233/SW-140134, https://madoc.bib. uni-mannheim.de/37476/ ↗
  12. li, J., Sun, A., Han, R., Li, C.: A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering PP, 1–1 (03 2020). https://doi.org/10.1109/TKDE.2020.2981314 Building and Evaluating Universal Named-Entity Recognition English corpus ↗
  13. Menezes, D.S., Savarese, P., Milidiú, R.L.: Building a massive corpus for named entity recognition using free open data sources (2019), http://arxiv.org/abs/1908.05758 ↗
  14. Ni, J., Dinu, G., Florian, R.: Weakly supervised cross-lingual named entity recognition via effective annotation and representation projection. CoRR abs/1707.02483 (2017), http://arxiv.org/abs/1707.02483 ↗
  15. Nothman, J., Ringland, N., Radford, W., Murphy, T., Curran, J.R.: Learning multilingual named entity recognition from wikipedia. Artif. Intell. 194, 151–175 (2013), http://dblp.uni-trier.de/db/journals/ai/ai194.html/NothmanRRMC13 ↗
  16. Ramshaw, L., Marcus, M.: Text chunking using transformation-based learning. In: Third Workshop on Very Large Corpora (1995), https://www.aclweb.org/anthology/W95-0107 ↗
  17. Sekine, S.: The Definition of Sekine’s Extended Named Entities. https://nlp.cs.nyu.edu/ene/version7_1_0Beng.html (07 2007), (Accessed on 28/02/2020) ↗
  18. Ševčı́ková, M., Žabokrtskỳ, Z., Krůza, O.: Named entities in Czech: annotating data and developing NE tagger. In: International Conference on Text, Speech and Dialogue. pp. 188–195. Springer (2007) find
  19. Weischedel, R., Palmer, M., Marcus, M., Hovy, E., Pradhan, S., Ramshaw, L., Xue, N., Taylor, A., Kaufman, J., Franchini, M., El-Bachouti, M., Belvin, R., Houston, A.: OntoNotes Release 5.0 (2013). https://doi.org/11272.1/AB2/MKJJ2R, https://hdl.handle.net/11272.1/AB2/MKJJ2R ↗
  20. Wikipedia: English Wikipedia — Wikipedia, the free encyclopedia. http://en.wikipedia.org/w/index.php?title=English%20Wikipedia&oldid=987449701 (2020), [Online; accessed 14-November-2020] ↗
  21. Yadav, V., Bethard, S.: A survey on recent advances in named entity recognition from deep learning models. CoRR abs/1910.11470 (2019), http://arxiv.org/abs/1910.11470 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alves-2022-building-evaluating-universal-named-entity/