Elamur Research⌕
☰
home / papers / 2020 / CLUENER2020: Fine-grained Named Entity Recognition Dataset and Benchmark for Chinese

CLUENER2020: Fine-grained Named Entity Recognition Dataset and Benchmark for Chinese

Paper · 2020 · Liang Xu, Yu tong, Qianqian Dong et al.

Summary

Chinese NER research has mostly relied on datasets with only a few coarse entity types (person, location, organization), and current models already score near 95 F1 on them. As a result, these datasets no longer separate strong systems well. The authors built CLUENER2020, a fine-grained Chinese NER benchmark with 10 entity types and 13,436 annotated news sentences drawn from THUCNews. Labels were first assigned automatically through vocabulary-based distant supervision and then corrected by hand. To make the data harder, the authors ran a 4-fold check with a small albert_tiny_zh model and discarded samples that every cross-fold model already got right. They also kept the category sizes roughly balanced. They tested BiLSTM-CRF, BERT and RoBERTa baselines and ran a two-stage human evaluation. The strongest model, RoBERTa-wwm-large, reaches only about 80 macro F1, and the human annotators scored lower than all pre-trained models. The dataset, baselines and a public leaderboard are released, giving the field a harder testbed for fine-grained Chinese entity recognition.

Key points

  • Entity types and splits: The 10 categories cover person names, positions, organizations, companies, government bodies, addresses, scenes, books, movies and games. The data splits into 10,748 train, 1,343 dev and 1,345 test samples, with an average sentence length of 37.4 characters.
  • Source data: Sentences come from THUCNews, which holds about 740k Sina News articles across 14 topics. Unlike Resume NER, whose largest class is 134 times its smallest, CLUENER2020 keeps all classes on a similar scale.
  • Model results (macro F1): BiLSTM-CRF scores 70.00, BERT-base 78.82 and RoBERTa-wwm-large 80.42. RoBERTa is best in every category, and address and scene are the hardest classes.
  • Human baseline: Three annotators first practised on 30 dev samples and then each labelled 100 test samples, averaging 63.41 F1.
  • Availability: Resources are released at github.com/CLUEbenchmark/CLUENER2020, along with a leaderboard on CLUEbenchmarks.com.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksNamed entity recognition · 200 works

Cite

Cite as Liang Xu, Yu tong, Qianqian Dong, Yixuan Liao, Cong Yu, Yin Tian, Weitang Liu, Lu Li, Caiquan Liu and Xuanwei Zhang. 2020. CLUENER2020: Fine-grained Named Entity Recognition Dataset and Benchmark for Chinese. arXiv:2001.04351.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2001.04351 · arXiv 2001.04351

@misc{xu2020cluener2020,
  title = {{CLUENER2020: Fine-grained Named Entity Recognition Dataset and Benchmark for Chinese}},
  author = {Liang Xu, Yu tong, Qianqian Dong, Yixuan Liao, Cong Yu, Yin Tian, Weitang Liu, Lu Li, Caiquan Liu and Xuanwei Zhang},
  year = {2020},
  doi = {10.48550/arxiv.2001.04351},
  eprint = {2001.04351},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2001.04351}
}

References

14 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  2. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019.
  3. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  4. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019.
  5. M Sun, J Li, Z Guo, Z Yu, Y Zheng, X Si, and Z Liu. Thuctc: an efficient chinese text classifier. GitHub Repository, https://github.com/thunlp/THUCTC (2016, accessed 17 May 2017). Google Scholar, 2016. ↗
  6. Jingbo Shang, Liyuan Liu, Xiang Ren, Xiaotao Gu, Teng Ren, and Jiawei Han. Learning named entity tagger using domain-specific dictionary. arXiv preprint arXiv:1809.03599, 2018.
  7. Gina-Anne Levow. The third international chinese language processing bakeoff: Word segmentation and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 108–117, 2006.
  8. Nanyun Peng and Mark Dredze. Named entity recognition for chinese social media with jointly trained embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 548–554, 2015.
  9. Hangfeng He and Xu Sun. F-score driven max margin neural network for named entity recognition in chinese social media. arXiv preprint arXiv:1611.04234, 2016. find
  10. Kerui Min, Chenggang Ma, Tianmei Zhao, and Haiyan Li. Bosonnlp: an ensemble approach for word segmentation and pos tagging. In Natural Language Processing and Chinese Computing, pages 520–526. Springer, 2015. find
  11. Yue Zhang and Jie Yang. Chinese ner using lattice lstm. 2018.
  12. Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991, 2015.
  13. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. Pre-training with whole word masking for chinese bert. arXiv preprint arXiv:1906.08101, 2019.
  14. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for language understanding. arXiv preprint arXiv:1907.12412, 2019.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2020-cluener2020-fine-grained-named/