Elamur Research⌕
☰
home / papers / 2015 / Boosting Named Entity Recognition with Neural Character Embeddings

Boosting Named Entity Recognition with Neural Character Embeddings

Paper · 2015 · Cícero dos Santos and Victor Guimarães

Summary

The paper asks whether named entity recognition can reach top accuracy without manually engineered features (gazetteers, dictionaries) or the output of POS taggers and chunkers. The authors apply CharWNN, a network that dos Santos and Zadrozny originally built for POS tagging. For each token it combines a word embedding with a character-level vector. The character vector comes from a convolution over character windows followed by max pooling. A window-based scorer assigns tag scores, and Viterbi decoding with learned tag-transition scores selects the best sequence. The hyperparameters are copied unchanged from the POS setup. Word embeddings are pre-trained on large unlabeled corpora and then fine-tuned during supervised training. The system is tested on Portuguese (HAREM I) and Spanish (SPA CoNLL-2002) and beats feature-heavy state-of-the-art systems on both. The gain is largest for Portuguese, where recall improves substantially. The work shows that character-level representations plus unsupervised word vectors can replace costly language-specific feature engineering. This matters for languages with few annotated resources.

Key points

  • Corpora: HAREM I (Portuguese, ten categories, trained on First HAREM and tested on MiniHAREM) and SPA CoNLL-2002 (Spanish).
  • Word embeddings:
    • Spanish: pre-trained on Spanish Wikipedia, about 450 million tokens after cleaning (lowercasing, digits mapped to 0, short sentences removed).
    • Portuguese: reused vectors trained on Portuguese Wikipedia, CETENFolha and CETEMPublico.
    • Character embeddings are randomly initialized, not pre-trained.
  • SPA CoNLL-2002: CharWNN reaches 82.21 F1 versus 81.39 for an AdaBoost system that relies on gazetteers, POS tags and other hand-built features.
  • HAREM I:
    • CharWNN scores 71.41 F1 in the total scenario and 77.93 in the selective scenario, against 63.56 and 70.72 for ETL_CMT.
    • Recall for ORG rises by about 21 points.
  • Ablations:
    • On HAREM I, word-only and char-only networks fall well behind the joint model. The joint model roughly ties a WNN that adds suffix and capitalization features.
    • Removing word-embedding pre-training costs 13.2 F1 on HAREM I and 4.3 F1 on Spanish.

Interesting ideas

  • I1. Convolution plus max pooling over characters. This turns words of any length into a fixed-size vector that captures shape and morphology. It could be useful for inflection-heavy languages, where suffixes and capitalization carry much of the signal.
  • I2. One configuration across tasks and languages. The authors kept a single set of hyperparameters across tasks (POS → NER) and languages. This is a cheap robustness check and makes a model easy to deploy where tuning data is scarce.
  • I3. Two-channel input design. Word vectors are trained on aggressively normalized text (lowercased, digits zeroed), while the character channel keeps the surface form. Splitting the work this way keeps the vocabulary compact without losing case or format cues.
  • I4. Pre-train, then fine-tune. Embeddings learned on raw text are still updated on labeled data, and the gain from pre-training grows as the labeled set shrinks. This argues for investing in unlabeled-corpus preparation when annotation budgets are small.
  • I5. Evaluation design. Results are reported under both an all-classes and a reduced-class scenario, with per-entity precision and recall. This shows where improvements actually come from (here, recall on ORG/PER/LOC) rather than hiding them in one number.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States85 works · 66,118 citations

Also on the byline: Universidade Federal Fluminense.

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2 Named entity recognition · 101 works Part-of-speech tagging · 91 works

Cite

Cite as Cícero dos Santos and Victor Guimarães. 2015. Boosting Named Entity Recognition with Neural Character Embeddings. Proceedings of the Fifth Named Entity Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w15-3904

@inproceedings{santos2015boosting,
  title = {{Boosting Named Entity Recognition with Neural Character Embeddings}},
  author = {Cícero {dos Santos} and Victor Guimarães},
  booktitle = {Proceedings of the Fifth Named Entity Workshop},
  year = {2015},
  doi = {10.18653/v1/w15-3904},
  url = {https://doi.org/10.18653/v1/w15-3904}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy). find
  2. Xavier Carreras, Lluı́s Màrques, and Lluı́s Padró. 2002. Named entity extraction using adaboost. In Proceedings of CoNLL-2002, pages 167–170. Taipei, Taiwan. find
  3. Yu Chen, You Ouyang, Wenjie Li, Dequan Zheng, and Tiejun Zhao. 2010. Using deep belief nets for chinese named entity categorization. In Proceedings of the Named Entities Workshop, pages 102–109. find
  4. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537. our card
  5. R. Collobert. 2011. Deep learning for efficient discriminative parsing. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS), pages 224–232. find
  6. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extraction (ace) program tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC-2004), Lisbo… our card
  7. Cı́cero Nogueira dos Santos and Ruy Luiz Milidiú, 2012. Entropy Guided Transformation Learning: Algorithms and Applications, chapter Named entity recognition, pages 51–58. Springer Briefs in Computer Science. Springer. find
  8. Cı́cero Nogueira dos Santos and Bianca Zadrozny. 2014a. Learning character-level representations for part-of-speech tagging. In Proceedings of the 31st International Conference on Machine Learning, JMLR: W&CP volume 32, Beijing, China. find
  9. Cı́cero Nogueira dos Santos and Bianca Zadrozny. 2014b. Training state-of-the-art portuguese POS taggers without handcrafted features. In Proceedings of the 11th International Conference Computational Processing of the Portuguese Language, pages 82–93, São Carlos, Brazil. find
  10. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 363–370. our card
  11. Yann Lecun, Lon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pages 2278–2324. find
  12. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of Workshop at International Conference on Learning Representations. our card
  13. Ruy Luiz Milidiú, Julio Cesar Duarte, and Roberto Cavalcante. 2007. Machine learning algorithms for portuguese named entity recognition. Revista Iberoamericana de Inteligencia Artificial, pages 67– 75. find
  14. Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 78–86, Ann Arbor, Michigan. our card
  15. Diana Santos and Nuno Cardoso. 2007. Reconhecimento de entidades mencionadas em português. Linguateca, Portugal. find
  16. Buzhou Tang, Hongxin Cao, Xiaolong Wang, Qingcai Chen, and Hua Xu. 2014. Evaluating word representation features in biomedical named entity recognition tasks. find
  17. BioMed Research International, 2014. find
  18. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Walter Daelemans and Miles Osborne, editors, Proceedings of CoNLL-2003, pages 142–147. Edmonton, Canada. our card
  19. Erik F. Tjong Kim Sang. 2002. Introduction to the conll-2002 shared task: Language-independent named entity recognition. In Proceedings of CoNLL-2002, pages 155–158. Taipei, Taiwan. our card
  20. A. J. Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13(2):260–269, April. find
  21. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. J. Lang. 1989. Phoneme recognition using time-delay neural networks. IEEE Transactions on Acoustics, Speech and Signal Processing, 37(3):328–339. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/santos-2015-boosting-named-entity-recognition-neural/