Elamur Research⌕
☰
home / papers / 2018 / BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages

BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages

Summary

Token-level embeddings handle rare and unseen words poorly. Subword methods address this, but BPE unit embeddings were usually trained for a single task and never released. The authors fill this gap with BPEmb. They ran the SentencePiece BPE implementation over every sufficiently large Wikipedia (lowercased, digits mapped to zero) and trained GloVe vectors for the resulting symbols. The release covers 275 languages and a grid of vocabulary sizes and dimensions. Because BPE works directly on raw symbol streams, no tokenizer is needed, which helps with scripts such as Chinese or Japanese.

To test usefulness, the authors used fine-grained entity typing of isolated mentions. They compared BPEmb with FastText, character embeddings and token embeddings, using three encoders and extensive hyper-parameter search. On English, BPEmb was the strongest subword option and matched FastText while being only a tiny fraction of its size. Across other languages the two were mostly on par, with a few gains and one large loss (Khmer). The resource makes compact multilingual subword vectors easy to obtain, including for low-resource and memory-limited settings.

Key points

  • BPEmb offers embeddings for 1000 to 200000 BPE merge operations (1k, 3k, 5k, 10k, 25k, 50k, 100k, 200k) and dimensions 25, 50, 100, 200 and 300. All are trained with GloVe on Wikipedia text extracted with WikiExtractor.
  • The test task assigns mentions to the 89 types of Gillick et al. (2014). Mentions come from Wikidata, and types come from mapping to Freebase, giving 3.4 million English instances with random 80,000/20,000 train/test samples. The encoders are averaging, RNN (LSTM) and CNN, tuned with a Tree-structured Parzen Estimator (≥1000 trials for English, ≥50 for others).
  • On English, BPE-RNN reached a mean of 0.624 ± 0.029 (max 0.65), against 0.617 ± 0.007 (max 0.63) for FastText-RNN. BPEmb matched FastText with about 11 MB (100k symbols, dim 25) versus roughly 6 GB (3 million vectors, dim 300).
  • Multilingual scores (FastText vs BPEmb):
    • English rose from 62.9 to 65.4.
    • Chinese rose from 71.0 to 72.0, and Japanese fell from 62.3 to 61.4, with FastText using pre-tokenized input (Stanford CoreNLP for Chinese, Kuromoji for Japanese).
    • Tibetan gained 3.5 and Lao gained 2.1.
    • Khmer dropped 8.9, which the authors attribute to inconsistent Unicode control-character handling across libraries.
  • Other findings:
    • Averaging was the weakest encoder.
    • RNNs slightly beat CNNs but trained much more slowly.
    • FastText showed the lowest variance across settings.
    • The evaluation ignores context and covers only a few of the 275 languages.

Interesting ideas

  • I1. Segmentation without a tokenizer. Running BPE on raw text sidesteps tokenizer quality for unsegmented scripts. This could suit mixed-script, noisy or poorly tooled languages, where a weak tokenizer would otherwise become the bottleneck.
  • I2. A grid of released variants. Publishing embeddings for many vocabulary sizes and dimensions lets users pick their own trade-off between size and quality. Other resource releases could do the same instead of shipping one "best" configuration.
  • I3. Score distributions instead of a single best run. Showing results across hundreds of hyper-parameter trials makes robustness visible. Here it showed that FastText is forgiving, while BPEmb needs careful tuning, which a single top score would hide.
  • I4. Mention-only entity typing as a probe. Typing a bare name like "Melfordshire" isolates what subword units carry about meaning. The data can be built automatically from a knowledge base, giving a cheap benchmark that scales to many languages.
  • I5. Small vocabulary and low dimension as a compact baseline. The paper shows that a small subword vocabulary with low-dimensional vectors can rival multi-gigabyte embeddings. This makes it worth trying such a setup first for on-device or resource-constrained applications.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

75 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 70 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{heinzerling2018bpemb,
  title = {{BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages}},
  author = {Heinzerling, Benjamin and Strube, Michael},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  doi = {10.63317/3eqtu58tsqay},
  url = {https://doi.org/10.63317/3eqtu58tsqay}
}

References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bergstra, J. S., Bardenet, R., Bengio, Y., and Kégl, B. (2011). Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems, pages 2546–2554. find
  2. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find
  3. Bollacker, K., Evans, C., Paritosh, P., Sturge, T., and Taylor, J. (2008). Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, Vancouver, B.C., Canada, 10–12 June 2008, pages 1247–12… find
  4. Chiu, J. and Nichols, E. (2016). Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, 4:357–370. find our card
  5. Clark, K. and Manning, C. D. (2016). Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, 7–12 August 2016. find our card
  6. Gillick, D., Lazic, N., Ganchev, K., Kirchner, J., and Huynh, D. (2014). Context-Dependent Fine-Grained Entity Type Tagging. ArXiv e-prints, December. find
  7. Ling, X. and Weld, D. S. (2012). Fine-grained entity recognition. In AAAI. find
  8. Luong, M.-T. and Manning, C. D. (2016). Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1054–1063, Berlin, Germany, August. Association fo… find our card
  9. Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. J., and McClosky, D. (2014). The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. Association for Computat… find our card
  10. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of the ICLR 2013 Workshop Track. find
  11. Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532– 1543, Doha, Qatar, October. Association for Computational Linguistics. find
  12. Reimers, N. and Gurevych, I. (2017). Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. CoRR, abs/1707.09861. find our card
  13. Schütze, H. (2017). Nonsymbolic text representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 785–796, Valencia, Spain, April. Association for Computational Linguistics. find
  14. Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August. Association for Computational Lin… find
  15. Shimaoka, S., Stenetorp, P., Inui, K., and Riedel, S. (2017). Neural architectures for fine-grained entity type classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1271–1280, Valencia, Spain, Apr… find
  16. Vrandečić, D. and Krötzsch, M. (2014). Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85. find
  17. Yaghoobzadeh, Y. and Schütze, H. (2017). Multi-level representations for fine-grained typing of knowledge base entities. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 578–589, Valencia, Spain, April. Asso… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/heinzerling-2018-bpemb-tokenization-free-pre-trained/