Elamur Research⌕
☰
home / papers / 2020 / FLAT: Chinese NER Using Flat-Lattice Transformer

FLAT: Chinese NER Using Flat-Lattice Transformer

Summary

Chinese NER is difficult because the text has no explicit word boundaries. Lexicon-matched character-word lattices help, but earlier lattice models such as lattice LSTM, LR-CNN and the GNN-based LGN and CGN are either recurrent or structurally complex. This makes them slow and poorly suited to GPU parallelism. The authors propose FLAT (Flat-LAttice Transformer), which rewrites the lattice as a flat set of spans. Each span is a character or a lexicon word, tagged with the positions of its first and last characters. The original lattice can be reconstructed from these spans. Relations between spans are captured by a relative position encoding built from four head/tail distances and fed into a relative self-attention variant. Only character outputs pass to a CRF layer. On Ontonotes 4.0, MSRA, Resume and Weibo, a single-layer FLAT beats other lexicon-based models in both F1 and inference speed. It also adds gains on top of BERT. The work shows that lattice information can be encoded through position encoding alone, without lattice-specific architectures.

Key points

  • Every token gets a head and a tail index. Four relative distances (head–head, head–tail, tail–head, tail–tail) are passed through sinusoidal encodings and a ReLU projection to form the span relation vector.
  • With the YJ lexicon, FLAT reaches F1 of 76.45 (Ontonotes), 94.12 (MSRA), 95.45 (Resume) and 60.32 (Weibo). On average it beats TENER by 1.72 F1 and lattice LSTM by 1.51 F1. With the LS lexicon, it beats CGN by 0.73.
  • Batch parallelism at batch size 16 speeds FLAT up 4.97×, compared with 2.1× for lattice LSTM. FLAT is the fastest of the compared models on Ontonotes.
  • Adding FLAT on top of BERT-wwm improves Ontonotes from 80.14 to 81.82 and MSRA from 94.95 to 96.09, with only marginal gains on the smaller Resume and Weibo sets.
  • Span F and Type Acc analyses indicate that the position encoding mainly improves entity boundary detection, while word embeddings help with type classification. Code is available at github.com/LeeSureman/Flat-Lattice-Transformer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition

Institutions

🎓Fudan UniversityShanghai · China45 works · 16 344 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.611

@inproceedings{li2020flat,
  title = {{FLAT: Chinese NER Using Flat-Lattice Transformer}},
  author = {Li, Xiaonan and Yan, Hang and Qiu, Xipeng and Huang, Xuanjing},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.611},
  url = {https://doi.org/10.18653/v1/2020.acl-main.611}
}

References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Oshin Agarwal, Yinfei Yang, Byron Wallace, and Ani Nenkova. 2020. Interpretability analysis for named entity recognition to understand system predictions and how they can improve. find
  2. Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Proces… find our card
  3. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-training with whole word masking for chinese BERT. CoRR, abs/1906.08101. find our card
  4. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. CoRR, abs/1901.02860. find
  5. Dennis Diefenbach, Vanessa Lopez, Kamal Singh, and Pierre Maret. 2018. Core techniques of question answering systems over knowledge bases: a survey. Knowledge and Information Systems, 55(3):529– 569. find
  6. Tao Gui, Ruotian Ma, Qi Zhang, Lujun Zhao, Yu-Gang Jiang, and Xuanjing Huang. 2019a. Cnn-based chinese ner with lexicon rethinking. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI’19, pages 4982–4988. AAAI Press. find
  7. Tao Gui, Yicheng Zou, Qi Zhang, Minlong Peng, Jinlan Fu, Zhongyu Wei, and Xuanjing Huang. 2019b. A lexicon-based graph neural network for Chinese NER. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langu… find
  8. Hangfeng He and Xu Sun. 2016. F-score driven max margin neural network for named entity recognition in chinese social media. CoRR, abs/1611.04234. find
  9. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, pages 282–289, San Francisco, CA, USA. Morgan Kaufm… find
  10. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, page… find our card
  11. Gina-Anne Levow. 2006. The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 108–117, Sydney, Australia. Association for Computational Linguistics. find our card
  12. Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du. 2018. Analogical reasoning on Chinese morphological and semantic relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 138–143, Melbourne, Australia. Asso… find
  13. Liyuan Liu, Jingbo Shang, Frank F. Xu, Xiang Ren, Huan Gui, Jian Peng, and Jiawei Han. 2017. Empower sequence labeling with task-aware neural language model. CoRR, abs/1709.04109. find our card
  14. Xue Mengge, Yu Bowen, Liu Tingwen, Wang Bin, Meng Erli, and Li Quangang. 2019. Porous lattice-based transformer encoder for chinese ner. find
  15. Nanyun Peng and Mark Dredze. 2015. Named entity recognition for Chinese social media with jointly trained embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 548–554, Lisbon, Portugal. Association for Computational Linguistics. find our card
  16. Matthias Sperber, Graham Neubig, Ngoc-Quan Pham, and Alex Waibel. 2019. Self-attentional models for lattice inputs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1185–1197, Florence, Italy. Association for Computational Linguistics. find
  17. Tomasz Stanislawek, Anna Wróblewska, Alicja Wójcicka, Daniel Ziembicki, and Przemyslaw Biecek. 2019. Named entity recognition - is there a glass ceiling? In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 624–633, Hong Kong, China. Association for Compu… find
  18. Dianbo Sui, Yubo Chen, Kang Liu, Jun Zhao, and Shengping Liu. 2019. Leverage lexical knowledge for Chinese named entity recognition via collaborative graph network. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on… find
  19. Tianxiang Sun, Yunfan Shao, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020. Learning sparse sharing architectures for multiple tasks. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces… find our card
  21. Ralph M Weischedel and Linguistic Data Consortium. 2013. Ontonotes release 5.0. Title from disc label. find
  22. Fengshun Xiao, Jiangtong Li, Hai Zhao, Rui Wang, and Kehai Chen. 2019. Lattice-based transformer encoder for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3090–3097, Florence, Italy. Association for Computational Linguis… find
  23. Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. 2019. Tener: Adapting transformer encoder for named entity recognition. find
  24. Jie Yang, Yue Zhang, and Fei Dong. 2017. Neural reranking for named entity recognition. CoRR, abs/1707.05127. batch lr -decay optimizer -momentum d m odel head FFN size embed dropout output dropout warmup 1e-3 0.05 SGD 0.9 160 8 480 0.5 0.3 10 (epoch) *Table 5: Hyper-parameters for Ontonotes and MSR… find
  25. Yue Zhang and Jie Yang. 2018. Chinese NER using lattice LSTM. CoRR, abs/1805.02023. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2020-flat-chinese-ner-flat-lattice/