Elamur Research⌕
☰
home / papers / 2014 / Lexicon Infused Phrase Embeddings for Named Entity Resolution

Lexicon Infused Phrase Embeddings for Named Entity Resolution

Summary

Strong named-entity recognition (NER) systems have relied on Brown clusters and hand-built lexicons. Brown clustering scales poorly to big vocabularies, and earlier neural embeddings had not beaten it. The authors extend the Skip-Gram model so that each phrase embedding must predict the phrase's context words and also whether the phrase belongs to each of 22 curated lexicon classes. These classes are built from Wikipedia categories and US Census data. The embeddings are trained on English Wikipedia plus RCV1 over a vocabulary of roughly 1M phrases. They are then used as features in a two-stage linear-chain CRF tagger. The lexicon-infused embeddings score slightly higher than plain Skip-Gram on the Mikolov et al. analogy test. On CoNLL 2003 they reach 90.90 test F1. That ties the best published score, which came from Lin and Wu (2009) using private query logs, and beats every previous system trained on public data. On Ontonotes 5.0, both embedding variants outperform Brown clusters. This was the first time neural-style word embeddings reached state-of-the-art NER results, at no extra training cost over standard Skip-Gram.

Key points

  • Training data and vocabulary: English Wikipedia (8M articles) plus RCV1 (946K documents).
    • Candidate phrases come from three sources: bigrams with PMI above 1000 (stopword bigrams removed), trigrams formed from high-PMI bigrams that share a token, and Wikipedia page titles.
    • This gives about 10M phrases, cut to the 1M most frequent.
  • Model settings: 50-dimensional embeddings, a context window of 21, and K = 22 lexicon classes. Only 0.01% of negative lexicon examples are sampled for training.
  • Analogy test (restricted to the top 30K words): Skip-Gram 29.89, Lex-0.05 30.37, Lex-0.01 30.72.
  • CoNLL 2003 test F1:
    • Baseline: 87.93
    • Gaz + Brown: 90.67
    • Gaz + Skip-gram: 90.33
    • Gaz + LexEmb: 90.90
    • Without gazetteer features, plain Skip-gram (89.68) trails Brown clusters (90.05).
  • Ontonotes 5.0 test F1:
    • Gaz + Brown: 82.05
    • Gaz + Skip-gram: 82.30
    • Gaz + LexEmb: 82.24
    • Both embedding variants beat Brown clusters here, but the lexicon-infused version is not the top one.

Interesting ideas

  • I1. Lexicons as an auxiliary training target. Instead of only using lexicons as features at tagging time, the authors use them as an extra prediction task while learning embeddings. Each lexicon gets its own logistic classifier, kept separate from the Huffman tree. This pulls together phrases that share a list, and the same trick could inject any curated resource (gazetteers, product catalogs, toponym lists) into embeddings cheaply.
  • I2. Heavy subsampling of negatives. Keeping just 0.01% of the non-member examples worked best for lexicon training. This suggests that a weak, sparse supervisory signal is enough to shape the vector space without overwhelming the language-modeling objective.
  • I3. Cheap phrase-vocabulary construction. The phrase list combines high-PMI bigrams, trigrams glued from bigrams that share a token, and Wikipedia page titles. This avoids repeated corpus passes and naturally captures multi-word entity names, which matters for languages and domains rich in multi-word place or organization names.
  • I4. Matching the representation to the downstream model. Log-linearly trained embeddings are argued to suit a linear CRF better than vectors from deep nonlinear networks. Smaller (50-dimensional) vectors helped NER even though larger ones scored better on analogies. The takeaway is to tune embeddings on the target task, not on intrinsic benchmarks alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition

Cited by

84 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 78 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-1609

@inproceedings{passos2014lexicon,
  title = {{Lexicon Infused Phrase Embeddings for Named Entity Resolution}},
  author = {Passos, Alexandre and Kumar, Vineet and McCallum, Andrew},
  booktitle = {Proceedings of the Eighteenth Conference on Computational Natural Language Learning},
  year = {2014},
  doi = {10.3115/v1/w14-1609},
  url = {https://doi.org/10.3115/v1/w14-1609}
}

References

26 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rie Kubota Ando and Tong Zhang. 2005. A high-performance semi-supervised learning method for text chunking. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 1–9. Association for Computational Linguistics. find
  2. Yoshua Bengio, Rejean Ducharme, and Pascal Vincent. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  3. Yoshua Bengio. 2008. Neural net language models. Scholarpedia, 3(1):3881. find
  4. Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, and Jenifer C Lai. 1992. Class-based n-gram models of natural language. Computational linguistics, 18(4):467–479. find
  5. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. find
  6. Paramveer Dhillon, Dean P Foster, and Lyle H Ungar. 2011. Multi-view learning of word embeddings via cca. In Advances in Neural Information Processing Systems, pages 199–207. find
  7. Paramveer Dhillon, Jordan Rodu, Dean Foster, and Lyle Ungar. 2012. Two step cca: A new spectral method for estimating vector models of words. arXiv preprint arXiv:1206.6403. — [arXiv:1206.6403](https://arxiv.org/abs/1206.6403) find
  8. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 999999:2121–2159. find
  9. David A Huffman. 1952. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 40(9):1098–1101. find
  10. Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing. find our card
  11. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. find
  12. David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. 2004. Rcv1: A new benchmark collection for text categorization research. The Journal of Machine Learning Research, 5:361–397. find our card
  13. Dekang Lin and Xiaoyun Wu. 2009. Phrase clustering for discriminative learning. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, pages 1030–1038. Association for Com… find
  14. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH, pages 1045–1048. find
  15. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  16. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. arXiv preprint arXiv:1310.4546. find
  17. Scott Miller, Jethran Guinness, and Alex Zamanian. 2004. Name tagging with word clusters and discriminative training. In HLT-NAACL, volume 4, pages 337–342. Citeseer. find our card
  18. Andriy Mnih and Geoffrey E Hinton. 2008. A scalable hierarchical distributed language model. In Advances in neural information processing systems, pages 1081–1088. find
  19. Frederic Morin and Yoshua Bengio. 2005. Hierarchical probabilistic neural network language model. In Proceedings of the international workshop on artificial intelligence and statistics, pages 246–252. find
  20. Fernando Pereira, Naftali Tishby, and Lillian Lee. 1993. Distributional clustering of english words. In Proceedings of the 31st annual meeting on Association for Computational Linguistics, pages 183–190. Association for Computational Linguistics. find
  21. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning, pages 147– 155. Association for Computational Linguistics. find our card
  22. Jun Suzuki and Hideki Isozaki. 2008. Semi-supervised sequential labeling and segmentation using gigaword scale unlabeled data. In ACL, pages 665–673. Citeseer. find
  23. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 142–147. Association for Computational Linguistics. find
  24. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394. Association for Computational Linguistics. find our card
  25. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2011. OntoNotes Release 4.0. Linguistic Data Consortium. find
  26. Tong Zhang and David Johnson. 2003. A robust risk minimization based named entity recognition system. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 204–207. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/passos-2014-lexicon-infused-phrase-embeddings-named/