Elamur Research⌕
☰
home / papers / 2016 / Multilingual Language Processing From Bytes

Multilingual Language Processing From Bytes

Paper · 2016 · Dan Gillick, Cliff Brunk, Oriol Vinyals et al.

Summary

The paper asks whether text annotation can work without the usual NLP pipeline of tokenization, sentence splitting and language-specific preprocessing. The authors propose Byte-to-Span (BTS), a sequence-to-sequence LSTM. It reads a text segment as raw UTF-8 bytes and then outputs annotations as a series of separate symbols: a start offset, a length and a label, ending with STOP. The input alphabet is only 256 byte values plus a few special symbols, so the model stays very small (about a million parameters). A single network can also be trained on many languages at once. The authors test it on POS tagging in 13 languages and on NER in English, German, Spanish and Dutch, using only the provided training data. The joint multilingual model matches or beats earlier systems that are limited to the same data. It also outperforms same-sized BTS models trained on each language alone. The work matters because it shows that bytes can act as a shared cross-lingual representation. This removes per-language engineering and yields compact standalone taggers.

Key points

  • On NER, the single BTS model trained on all four languages scores 86.50 F1 (en), 76.22 (de), 82.95 (es) and 82.84 (nl). The per-language BTS* models score lower: 84.57, 72.08, 81.83 and 78.08.
  • Byte-dropout randomly replaces input bytes with a DROP symbol. Combined with standard dropout, it raises average NER F1 from 74.75 (vanilla) to 78.76 (dropout) and then to 82.13. Average POS accuracy over 13 languages rises from 94.78 to 95.35 and then to 95.85.
  • Depth helps more than width. With 320 units, NER F1 climbs from 76.15 with one layer to 82.13 with four layers. Doubling the width to 640 at depth 4 adds almost nothing (82.18).
  • The encoder and decoder share one set of LSTM weights without losing quality. Adding attention did not improve NER, although training converged faster.
  • A tSNE plot of the hidden state before NER label prediction shows clusters grouped by label rather than by language.

Interesting ideas

  • I1. Bytes as a universal input. Feeding raw UTF-8 bytes avoids tokenizers and per-language vocabularies. This could suit mixed-script or code-mixed text, such as Ukrainian–Russian–English data, where tokenization is messy.
  • I2. Split structured outputs into a sequence. Predicting start, length and label as three consecutive outputs keeps the output vocabulary small, and the authors report better and faster training than with the combined triple space. The same trick could apply to other span-style tasks that would otherwise need a BIO scheme or a huge label set.
  • I3. Input-noise regularization (byte-dropout). Corrupting random input symbols works like adding noise to audio or blurring images, and it gave large gains here. It is cheap to add to any character- or byte-level model.
  • I4. Diagnose shared representations through architecture and visualization. The authors compare the gains from depth versus width and inspect the hidden states with tSNE to check whether a multilingual model separates languages or learns shared abstractions. This is a reusable way to audit multilingual models.
  • I5. Be aware of the precision/recall asymmetry. A model that only emits spans lets you raise precision by thresholding the product of the span's probabilities, but offers no obvious way to raise recall. This is worth knowing before choosing a generative span output for tasks where recall matters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 465 works

Cite

Cite as Dan Gillick, Cliff Brunk, Oriol Vinyals and Amarnag Subramanya. 2016. Multilingual Language Processing From Bytes. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1155

@inproceedings{gillick2016multilingual,
  title = {{Multilingual Language Processing From Bytes}},
  author = {Dan Gillick, Cliff Brunk, Oriol Vinyals and Amarnag Subramanya},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1155},
  url = {https://doi.org/10.18653/v1/n16-1155}
}

References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. The Journal of Machine Learning Research, 6:1817–1853. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  3. Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. 2015. End-to-end attention-based large vocabulary speech recognition. arXiv preprint arXiv:1508.04395. — [arXiv:1508.04395](https://arxiv.org/abs/1508.04395) find
  4. Miguel Ballesteros, Chris Dyer, and Noah A Smith. 2015. Improved transition-based parsing by modeling characters instead of words with lstms. arXiv preprint arXiv:1508.00657. Xavier Carreras, Lluı́s Màrques, and Lluı́s Padró. 2002.
  5. Named entity extraction using adaboost. In Proceedings of CoNLL-2002, pages 167–170. find
  6. Taipei, Taiwan. William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. 2015. Listen, attend and spell. arXiv preprint arXiv:1508.01211. find
  7. Rohan Chitnis and John DeNero. 2015. Variable-length word encodings for neural translation models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2088– 2093. find
  8. Kenneth Ward Church. 1993. Char align: a program for aligning parallel texts at the character level. In Proceedings of the 31st annual meeting on Association for Computational Linguistics, pages 1–8. Association for Computational Linguistics. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karl… find
  9. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. arXiv preprint arXiv:1511.01432. Cıcero dos Santos, Victor Guimaraes, RJ Niterói, and Rio de Janeiro. 2015. Boosting named entity recognition with neural character embeddings. In Proceedings of NEWS 2015 The Fifth Named Entities W…
  10. Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015. Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser. In 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (V… find
  11. Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent. 2009. The difficulty of training deep architectures and the effect of unsupervised pre-training. In International Conference on artificial intelligence and statistics, pages 153–160. Florian Eyben, Martin Wöllme… find
  12. ASRU 2009. IEEE Workshop on, pages 376–380. find
  13. IEEE. Radu Florian, Abe Ittycheriah, Hongyan Jing, and Tong Zhang. 2003. Named entity recognition through classifier combination. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 168–171. Association for Computational Linguistics.
  14. Georgia Frantzeskou, Efstathios Stamatatos, Stefanos Gritzalis, and Sokratis Katsikas. 2006. Effective identification of source code authors using byte-level information. In Proceedings of the 28th international conference on Software engineering, pages 893–896. find
  15. ACM. Alex Graves and Navdeep Jaitly. 2014. Towards end-to-end speech recognition with recurrent neural networks. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 1764–1772. find
  16. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computati… find
  17. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
  18. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2015. Character-aware neural language models. arXiv preprint arXiv:1508.06615.
  19. Dan Klein, Joseph Smarr, Huy Nguyen, and Christopher D Manning. 2003. Named entity recognition with character-level models. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 180–183. Association for Computational Linguistics. find
  20. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. Wang Ling, Tiago Luı́s, Luı́s Marujo, Ramón Fernandez Astudillo, Silvio Amir, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015. Finding functi…
  21. Christopher D Manning. 2011. Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In Computational Linguistics and Intelligent Text Processing, pages 171–189.
  22. Springer. Tetsuji Nakagawa. 2004. Chinese and japanese word segmentation using word-level and character-level information. In Proceedings of the 20th international conference on Computational Linguistics, page 466. Association for Computational Linguistics. find
  23. Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R Curran. 2013. Learning multilingual named entity recognition from wikipedia. Artificial Intelligence, 194:151–175. find
  24. Alexandre Passos, Vineet Kumar, and Andrew Mc-Callum. 2014. Lexicon infused phrase embeddings for named entity resolution. arXiv preprint arXiv:1404.5367.
  25. Fuchun Peng, Dale Schuurmans, Shaojun Wang, and Vlado Keselj. 2003. Language independent authorship attribution using character level language models. In Proceedings of the tenth conference on European chapter of the Association for Computational Linguistics-Volume 1, pages 267–274. Association for… find
  26. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011. A universal part-of-speech tagset. arXiv preprint arXiv:1104.2086.
  27. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  28. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329.
  29. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, pages 649–657.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gillick-2016-multilingual-language-processing-bytes/