Elamur Research⌕
☰
home / papers / 2018 / NCRF++: An Open-source Neural Sequence Labeling Toolkit

NCRF++: An Open-source Neural Sequence Labeling Toolkit

Paper · 2018 · Jie Yang and Yue Zhang

Summary

Neural sequence labeling models such as LSTM-CRF had become state of the art for NER, chunking and POS tagging. However, released code was usually tied to one architecture and one task, and changing it took substantial programming. NCRF++ is a PyTorch toolkit that fills this gap, in the same way CRF++ did for statistical CRFs. It reads CoNLL-format data and organizes models into three layers: a character sequence layer (RNN/GRU/LSTM or CNN), a word sequence layer (stackable RNN or CNN) and an inference layer (softmax or CRF). Users assemble models by editing a configuration file, and they can add hand-crafted features as embedded lookup tables. The toolkit computes in batches on GPU and extends Viterbi decoding to return n-best label sequences with probabilities. Re-implemented models were tested on CoNLL 2003 NER, CoNLL 2000 chunking and POS tagging. With nearly identical hyperparameters across all runs, they match or slightly exceed published results. This makes the toolkit a practical base for reproducing and extending sequence labeling work.

Key points

  • The best configuration, CCNN+WLSTM+CRF, reaches 91.35 F1 on CoNLL 2003 NER and 95.06 F1 on CoNLL 2000 chunking. Its POS accuracy of 97.46 is slightly below the best reported 97.55.
  • Word-level CNN encoders consistently trail word-level LSTM encoders.
  • Adding character information gives a clear gain: NER rises from 89.45 to about 91.2–91.35 with the LSTM word encoder. Character LSTM and character CNN help by similar amounts.
  • Hand-crafted [POS] and [Cap] features, each embedded in 10 dimensions, lift the NER baseline from 89.15 F1 to 90.59 when combined. Automatic character features do better still.
  • With 10-best decoding, oracle entity F1 rises from 91.35 to 97.47 and token accuracy from 98.00 to 99.39.
  • On a GTX 1080, decoding exceeds 2000 sentences/s, leveling off around batch size 100, while training exceeds 1000 sentences/s.

Interesting ideas

  • I1. Configuration-only model assembly. A complete LSTM-CRF with a char CNN and extra features fits in about ten configuration lines. Declarative specs like this make ablation grids and reproducibility much cheaper for any modeling team.
  • I2. Strict layer boundaries for plug-ins. A custom module only has to map character indexes to a representation, and everything else stays configurable. This interface discipline lets new encoders, for example for Ukrainian morphology, be inserted without touching the rest of the code.
  • I3. Discrete features as small learned embeddings. Tags such as POS or capitalization become low-dimensional vectors concatenated with word vectors, optionally pretrained. This is a simple way to inject linguistic knowledge into neural taggers.
  • I4. N-best output with oracle evaluation. Reporting how often the correct labeling appears among the top-k candidates shows how much headroom remains for downstream rerankers or pipelines. It is useful wherever a later stage can choose among candidates.
  • I5. One fixed hyperparameter set across tasks. Keeping settings nearly constant across all experiments is a robustness check that avoids per-task tuning inflating results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Singapore University of Technology and DesignSingapore · Singapore33 works · 10,760 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Part-of-speech tagging · 91 works

Cite

Cite as Jie Yang and Yue Zhang. 2018. NCRF++: An Open-source Neural Sequence Labeling Toolkit. Proceedings of ACL 2018, System Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-4013

@inproceedings{yang2018ncrf,
  title = {{NCRF++: An Open-source Neural Sequence Labeling Toolkit}},
  author = {Jie Yang and Yue Zhang},
  booktitle = {Proceedings of ACL 2018, System Demonstrations},
  year = {2018},
  doi = {10.18653/v1/p18-4013},
  url = {https://doi.org/10.18653/v1/p18-4013}
}

References

23 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via Bi-directional LSTM-CNNs-CRF. In ACL. volume 1, pages 1064–1074.
  2. Jason Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association Naoaki Okazaki. 2007. Crfsuite: a fast implementation for Computational Linguistics 4:357–370. of conditional random fields (crfs) . https://transacl.org/ojs/index.php/tacl/artic… ↗
  3. Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised se-Ronan Collobert, Jason Weston, Léon Bottou, Michael quence tagging with bidirectional language models. Karlen, Koray Kavukcuoglu, and Pavel Kuksa. In ACL. volume 1, pages 1756–1765. 2011. Natural language… find
  4. Nguyen. 2004. Flexcrfs: Flexible conditional random fields. find
  5. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In In-Lev Ratinov and Dan Roth. 2009. Design challenges ternational Conference on Artificial Intelligence and and misconceptions in named entity recognition. In Statistics. pages 315–323. CoNLL. pages 147–… find
  6. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780. find
  7. Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. In EMNLP. pages 338–348.
  8. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991 .
  9. Fei Sha and Fernando Pereira. 2003. Shallow parsing with conditional random fields. In NAACL-HLT. pages 134–141. find
  10. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning. pages 448–456. find
  11. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15(1):1929–1958. find
  12. S Sathiya Keerthi and Sellamanickam Sundararajan. 2007. Crf versus svm-struct for sequence labeling. Yahoo Research Technical Report . find
  13. Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017. Fast and accurate entity recognition with iterated dilated convolutions. In EMNLP. pages 2670–2680.
  14. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In International Conference on Machine Learning. volume 1, pages 282–289. find
  15. Erik F Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the conll-2000 shared task: Chunking. In Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning-Volume 7. pages 127–132.
  16. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In NAACL-HLT. pages 260–270.
  17. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In HLT-NAACL. pages 142–147.
  18. Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. 1989. Backpropagation applied to handwritten zip code recognition. Neural computation 1(4):541–551. find
  19. Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE transactions on Information Theory 13(2):260–269. find
  20. Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In EMNLP. pages 1520– 1530.
  21. Jie Yang, Shuailong Liang, and Yue Zhang. 2018. Design challenges and misconceptions in neural sequence labeling. In COLING.
  22. Zhilin Yang, Ruslan Salakhutdinov, and William W Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. In International Conference on Learning Representations. find
  23. Liyuan Liu, Jingbo Shang, Frank Xu, Xiang Ren, Huan Gui, Jian Peng, and Jiawei Han. 2018. Empower sequence labeling with task-aware neural language model. In AAAI.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2018-ncrf-open-source-neural-sequence/