Elamur Research
☰
home / papers / 2016 / OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles

OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/L16-1147.pdf

Summary

The paper introduces OpenSubtitles2016, a much larger and cleaner version of the OpenSubtitles parallel corpus collection. The authors built it from a dump of the OpenSubtitles.org database, about 3.36 million subtitle files in more than 60 languages. Their pipeline has several stages:

  • It detects each file's character encoding using per-language candidate lists and the chardet library.
  • It turns timed subtitle blocks into tokenised sentences with language-specific rules.
  • It fixes OCR mistakes and misplaced accents using a noisy-channel model with a bigram language model.
  • It stores each subtitle as an XML file.

Subtitles for the same movie or episode are then aligned across languages by how much their timings overlap. When several alternative subtitles exist, metadata-based quality scores choose the best pair. Alternative subtitles in the same language are also aligned with each other.

The result is 1689 bitexts covering 60 languages. The data grew about 39% over the previous release, and Moses SMT systems trained on it gain BLEU for nearly all tested pairs. This makes it one of the largest freely available parallel resources for conversational language (on OPUS).

Key points

  • Scale: 2.6 billion sentences (17.2 billion tokens) from about 2.8M converted subtitles, covering 152,939 distinct movies and TV episodes by IMDb identifier. New languages are Breton, Esperanto, Georgian and Tagalog.
  • Largest bitext: English–Spanish, with about 62.2K aligned documents and 49.2M sentence pairs.
  • OCR and accent correction: applied to 11 languages, fixing 9.04 million words, about 3.2 corrections per subtitle on average.
  • Extrinsic evaluation: Moses phrase-based SMT with MERT tuning on 18 language pairs. The 2016 data mostly beats the 2012+2013 release, for example 35.49 → 39.15 BLEU on the top pair.
  • Ukrainian: relatively small, with about 1.0K subtitle files, 0.9K covered IMDbs, 0.8M sentences and 5.4M tokens.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

441 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 423 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lison2016opensubtitles2016,
  title = {{OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles}},
  author = {Lison, Pierre and Tiedemann, Jörg},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  url = {https://aclanthology.org/L16-1147/}
}

References

12 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aziz, W., de Sousa, S. C. M., and Specia, L. (2012). Cross-lingual sentence compression for subtitles. In 16th Annual Conference of the European Association for Machine Translation (EAMT 2012), pages 103–110, Trento, Italy. find
  2. Brants, T. and Franz, A. (2006). Web 1T 5-gram corpus version 1. Technical report, Google Research. find
  3. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C. J., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the A… find
  4. Li, S. and Momoi, K. (2001). A composite approach to language/encoding detection. In 19th International Unicode Conference (IUC 2001), San Jose, California. find
  5. Lui, M. and Baldwin, T. (2012). Langid.py: An off-the-shelf language identification tool. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012 System Demonstrations), pages 25–30, Jeju Island, Korea. find our card
  6. Neubig, G., Nakata, Y., and Mori, S. (2011). Pointwise prediction for robust, adaptable japanese morphological analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL 2011), pages 529–533, Portland, Oregon, USA. find
  7. Och, F. J. (2003). Minimum Error Rate Training in Statistical Machine Translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics (ACL 2003), pages 160–167, Sapporo, Japan. find
  8. Tiedemann, J. (2007). Improved sentence alignment for movie subtitles. In Proceedings of the Conference on Recent Advances in Natural Language Processing (RANLP 2007), Borovets, Bulgaria. find
  9. Tiedemann, J. (2008). Synchronizing translated movie subtitles. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008), pages 1902–1906, Marrakesh, Marocco. find
  10. Tiedemann, J. (2012). Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC 2012), pages 2214–2218, Istanbul, Turkey. find
  11. Tiedemann, J. (2016). Finding alternative translations in a large corpus of movie subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), Portorož, Slovenia. find
  12. Zhang, S., Ling, W., and Dyer, C. (2014). Dual subtitles as parallel corpora. In Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC 2014), pages 1869–1874, Reykjavik, Iceland. find