Elamur Research⌕
☰
home / papers / 2015 / Named Entity Recognition for Chinese Social Media with Jointly Trained Embeddings

Named Entity Recognition for Chinese Social Media with Jointly Trained Embeddings

Summary

Peng and Dredze address named entity recognition on Chinese social media, which had not been studied before. Earlier Chinese NER work used formal text such as news, and social media NER work focused mostly on English. They built a new corpus of Sina Weibo posts, annotated for both name and nominal mentions under the DEFT ERE guidelines. Their baseline is a character-level CRF that re-implements Mao et al. (2008). To it they add three kinds of embeddings learned from about 2.26 million unlabeled Weibo posts: word, character, and character+position. Each embedding type is tested with and without fine-tuning. They also propose a joint objective that optimizes the CRF likelihood and a skip-gram language model at once, with shared embeddings. On test data, the joint model with character+position embeddings scores best, gaining up to 9 F1 points over the baseline. Absolute scores remain low, around 40–44 F1. The authors note that the gap between social media and formal text is much wider for Chinese than for English, so this is a hard and open problem.

Key points

  • Corpus: 1,890 Weibo messages from November 2013 to December 2014, labeled with person, organization, location and geo-political entity types.
    • Annotation covers both name and nominal mentions.
    • The data is split into 5 folds for training, 1 for development and 1 for test.
    • Code and data are released as hltcoe/golden-horse.
  • Baseline: a CRF that tags characters with BIOSE labels, using character unigram and bigram features in a window of 5.
    • The re-implementation leaves out Mao et al.'s first-pass mention detector and gazetteer features.
    • It reaches 88.63 F1 on SIGHAN 2008.
    • Retrained Stanford NER serves as a second baseline.
  • Embedding vocabularies: 179,809 word embeddings (segmented with Jieba), 10,912 character embeddings, and 24,818 character+position embeddings.
  • Test F1, name mentions: baseline 35.00, character+position 43.48, joint model 44.09.
  • Test F1, name+nominal mentions: baseline 34.18, joint model 40.38.
  • Error analysis: 150 sampled errors were inspected by hand.
    • About 43% were annotation mistakes, mostly mentions the annotators missed.
    • Person names accounted for 9% and transliterated foreign names for 7%.

Interesting ideas

  • I1. Position-tagged character embeddings: each character gets a separate vector depending on whether it is the first, second, etc. character of its segmented word. This sits between noisy word-level and ambiguous character-level units. The same idea could suit other languages where segmentation is unreliable, or subword models for morphologically rich languages.
  • I2. Joint training instead of post-hoc fine-tuning: the task loss and a skip-gram loss are summed, with a weight balancing them. Embeddings can then adapt to the task without drifting away from the raw-text statistics. Embeddings of words that never appear in the labeled set also benefit, which is valuable for out-of-vocabulary-heavy domains like social media.
  • I3. Biased sampling for annotation: posts were chosen if they contained at least three words outside a list of common words, rather than sampled at random. This raises entity density per annotated message. It is a cheap trick for any low-resource annotation project where target phenomena are rare.
  • I4. Learning curves to measure embedding value: comparing systems with and without embeddings at training sizes from 200 to 1,400 instances showed that the gains are largest when labeled data is scarce. This is a useful evaluation habit for judging pretrained resources in new languages or domains.
  • I5. Manual error audits on low scores: a hand-checked error sample revealed that a large share of "errors" were annotation gaps. This is a reminder to verify gold data quality before trusting low numbers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition

Cited by

87 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 83 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1064

@inproceedings{peng2015named,
  title = {{Named Entity Recognition for Chinese Social Media with Jointly Trained Embeddings}},
  author = {Peng, Nanyun and Dredze, Mark},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1064},
  url = {https://doi.org/10.18653/v1/d15-1064}
}

References

43 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacqueline Aguilar, Charley Beller, Paul McNamee, Benjamin Van Durme, Stephanie Strassel, Zhiyi Song, and Joe Ellis. 2014. A comparison of the events and relations across ACE, ERE, TAC-KBP, and FrameNet annotation standards. In ACL Workshop: EVENTS. find our card
  2. Razvan C Bunescu and Raymond J Mooney. 2005. A shortest path dependency kernel for relation extraction. In Empirical Methods in Natural Language Processing (EMNLP), pages 724–731. Association for Computational Linguistics. find our card
  3. Chris Callison-Burch and Mark Dredze. 2010. Creating speech and language data with Amazon’s Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Mechanical Turk. find
  4. Xinxiong Chen, Lei Xu, Zhiyuan Liu, Maosong Sun, and Huanbo Luan. 2015. Joint learning of character and word embeddings. In International Joint Conference on Artificial Intelligence (IJCAI’15). find
  5. Colin Cherry and Hongyu Guo. 2015. The unreasonable effectiveness of word representations for twitter named entity recognition. In North America Chapter of Association for Computational Linguistics (NAACL). Association for Computational Linguistics. find
  6. Michael Collins and Yoram Singer. 1999. Unsupervised models for named entity classification. In Empirical Methods in Natural Language Processing (EMNLP), pages 100–110. Citeseer. find our card
  7. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In International Conference on Machine Learning (ICML), pages 160–167. ACM. find
  8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. find our card
  9. Mark Dredze, Paul McNamee, Delip Rao, Adam Gerber, and Tim Finin. 2010. Entity disambiguation for knowledge base population. In Conference on Computational Linguistics (Coling). find
  10. Huiming Duan, Zhifang Sui, Ye Tian, and Wenjie Li. 2012. The cips-sighan clp 2012 chineseword segmentation onmicroblog corpora bakeoff. In Second CIPS-SIGHAN Joint Conference on Chinese Language Processing, pages 35–40, Tianjin, China, December. Association for Computational Linguistics. find
  11. David Etter, Francis Ferraro, Ryan Cotterell, Olivia Buzek, and Benjamin Van Durme. 2013. Nerit: Named entity recognition for informal text. Technical report, Technical Report 11, Human Language Technology Center of Excellence, Johns Hopkins University, July. find
  12. Xiaoshan Fang, Jianfeng Gao, and Huanye Sheng. 2004. A semi-supervised approach to build annotated corpus for chinese named entity recognition. In Oliver Streiter and Qin Lu, editors, ACL SIGHAN Workshop 2004, pages 129–133, Barcelona, Spain, July. Association for Computational Linguistics. find
  13. Tim Finin, William Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. Annotating named entities in twitter data with crowdsourcing. In NAACL Workshop on Creating Speech and Language Data With Mechanical Turk. find
  14. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Association for Computational Linguistics (ACL), pages 363–370. Association for Computational Linguistics. find our card
  15. Hege Fromreide, Dirk Hovy, and Anders Søgaard. 2014. Crowdsourcing and annotating NER for Twitter\# drift. In LREC. find
  16. Zhengyan He, Houfeng Wang, and Sujian Li. 2012a. The task 2 of cips-sighan 2012 named entity recognition and disambiguation in chinese bakeoff. In Second CIPS-SIGHAN Joint Conference on Chinese Language Processing, pages 108–114, Tianjin, China, December. Association for Computational Linguistics. find
  17. Zhengyan He, Houfeng Wang, and Sujian Li. 2012b. The task 2 of cips-sighan 2012 named entity recognition and disambiguation in chinese bakeoff. In Second CIPS-SIGHAN Joint Conference on Chinese Language Processing, pages 108–114, Tianjin, China, December. Association for Computational Linguistics. find
  18. Guangjin Jin and Xiao Chen. 2008. The fourth international chinese language processing bakeoff: Chinese word segmentation, named entity recognition and chinese pos tagging. In Sixth SIGHAN Workshop on Chinese Language Processing, page 69. Citeseer. find
  19. Chenliang Li, Jianshu Weng, Qi He, Yuxia Yao, Anwitaman Datta, Aixin Sun, and Bu-Sung Lee. 2012. Twiner: Named entity recognition in targeted twitter stream. In SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’12, pages 721–730, New York, NY, USA. ACM. find
  20. Wang Ling, Guang Xiang, Chris Dyer, Alan Black, and Isabel Trancoso. 2013. Microblogs as parallel corpora. In Association for Computational Linguistics (ACL). Association for Computational Linguistics. find
  21. Linguistics Data Consortium. 2014. DEFT ERE Annotation Guidelines: Entities. find
  22. Xiaohua Liu, Shaodian Zhang, Furu Wei, and Ming Zhou. 2011. Recognizing named entities in tweets. In Association for Computational Linguistics (ACL), pages 359–367. Association for Computational Linguistics. find
  23. Xiaohua Liu, Ming Zhou, Furu Wei, Zhongyang Fu, and Xiangyang Zhou. 2012. Joint inference of named entity recognition and normalization for tweets. In Association for Computational Linguistics (ACL), ACL ’12, pages 526–535, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  24. Xiaodong Liu, Kevin Duh, Yuji Matsumoto, and Tomoya Iwakura. 2014. Learning character representations for chinese word segmentation. In NIPS 2014 Workshop on Modern Machine Learning and Natural Language Processing. find
  25. Xinnian Mao, Yuan Dong, Saike He, Sencheng Bao, and Haila Wang. 2008. Chinese word segmentation and named entity recognition based on conditional random fields. In IJCNLP, pages 90–93. find
  26. Andrew McCallum and Wei Li. 2003. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In North America Chapter of Association for Computational Linguistics (NAACL). find our card
  27. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Neural Information Processing Systems (NIPS), pages 3111–3119. find
  28. David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3–26. find
  29. Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. CoRR, abs/1404.5367. find our card
  30. Wenzhe Pei, Tao Ge, and Chang Baobao. 2014. Max-margin tensor neural network for chinese word segmentation. In Association for Computational Linguistics (ACL). find
  31. Siyu Qiu, Qing Cui, Jiang Bian, Bin Gao, and Tie-Yan Liu. 2014. Co-learning of word representations and morpheme representations. In Conference on Computational Linguistics (Coling). find
  32. Lev Ratinov, Dan Roth, Doug Downey, and Mike Anderson. 2011. Local and global algorithms for disambiguation to wikipedia. In Association for Computational Linguistics (ACL), pages 1375–1384. Association for Computational Linguistics. find
  33. Alan Ritter, Sam Clark, Oren Etzioni, et al. 2011. Named entity recognition in tweets: an experimental study. In Empirical Methods in Natural Language Processing (EMNLP), pages 1524–1534. Association for Computational Linguistics. find our card
  34. Yaming Sun, Lei Lin, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2014. Radical-enhanced chinese character embedding. In Neural Information Processing Systems (NIPS), pages 279–286. Springer. find
  35. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Association for Computational Linguistics (ACL), pages 384–394. Association for Computational Linguistics. find our card
  36. Xianxiang Yang, Heyan Huang, Xin Xin, Quanchao Liu, and Xiaochi Wei. 2014. Domain-specific product named entity recognition from chinese microblog. In Computational Intelligence and Security (CIS), 2014 Tenth International Conference on, pages 218–222. IEEE. find
  37. Kaisheng Yao, Baolin Peng, Geoffrey Zweig, Dong Yu, Xiaolong Li, and Feng Gao. 2014. Recurrent conditional random field for language understanding. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, pages 4077–4081. IEEE. find
  38. Mo Yu and Mark Dredze. 2014. Improving lexical embeddings with semantic knowledge. In Association for Computational Linguistics (ACL), pages 545–550. find our card
  39. Mo Yu and Mark Dredze. 2015. Learning composition models for phrase embeddings. Transactions of the Association for Computational Linguistics, 3:227– 242. find
  40. Suxiang Zhang, Ying Qin, Juan Wen, and Xiaojie Wang. 2006. Word segmentation and named entity recognition for sighan bakeoff3. In Fifth SIGHAN Workshop on Chinese Language Processing, pages 158–161, Sydney, Australia, July. Association for Computational Linguistics. find
  41. Longkai Zhang, Houfeng Wang, Xu Sun, and Mairgup Mansur. 2013. Exploring representations from unlabeled data with co-training for chinese word segmentation. In Empirical Methods in Natural Language Processing (EMNLP). find
  42. Xiaoqing Zheng, Hanyang Chen, and Tianyu Xu. 2013. Deep learning for chinese word segmentation and pos tagging. In Empirical Methods in Natural Language Processing (EMNLP), pages 647–657. find
  43. Xiaodan Zhu, Mu Li, Jianfeng Gao, and Chang-Ning Huang. 2003. Single character chinese named entity recognition. In Second SIGHAN Workshop on Chinese Language Processing, pages 125–132, Sapporo, Japan, July. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2015-named-entity-recognition-chinese-social/