Elamur Research⌕
☰
home / papers / 2018 / OpenSubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora

OpenSubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora

Summary

Subtitles exist in huge numbers and many languages, which makes them attractive for building parallel corpora. But subtitle translations are often condensed or loosely rendered rather than literal, so the resulting sentence alignments are noisier than those from other bitext sources. The authors release OpenSubtitles2018, built from a full OpenSubtitles database dump. After cleanup it contains 3.7M subtitle files in 60 languages, roughly 30% more sentences than the 2016 release.

The pipeline includes several steps:

  • format conversion
  • sentence segmentation
  • tokenisation
  • OCR error correction
  • IMDb metadata enrichment
  • time-overlap alignment

The main addition is a quality score attached to every sentence pair. For a sample of pairs, IBM Model 1 lexical translation probabilities serve as a proxy target. A small feedforward network then learns to predict this target from cheap, language-independent features. It reaches an RMSE of 0.070 and R² of 0.47, and manual inspection confirms that low-scored pairs are typically misalignments.

NMT experiments that filter training data at a fixed threshold gave mixed results. Users can still prune noisy pairs from this very large open multilingual resource according to their own needs.

Key points

  • Release size: 3.4G sentences and 22.2G tokens across 60 languages and 1782 language pairs.
    • The sources are 3.98M raw files, of which about 180K were dropped for formatting or metadata errors.
    • The files cover about 208K movies/TV episodes. Of the IMDb titles, 69% come with subtitles in two or more languages, while 29% have them in ten or more.
  • Ukrainian: about 1.6K subtitle files (up from 1.0K in 2016), covering 1.4K IMDb titles, with 1.3M sentences and 7.9M tokens.
  • Preprocessing and metadata:
    • The noisy-channel OCR corrector, based on Google N-grams, fixed over 9M tokens, about 4M of them in English.
    • New metadata flags mark subtitles for hearing-impaired audiences and subtitles produced by machine translation.
    • Alternative (non-top-ranked) subtitle alignments are now kept in separate files.
  • Model comparison for predicting the quality score:
    • two-hidden-layer feedforward NN (dim 100): R² 0.470
    • ridge regression: R² 0.356
    • gradient boosting: R² 0.224
  • NMT test: HNMT with BPE on WMT 2017 language pairs, filtering at score 0.6. BLEU differences were small and inconsistent, both on held-out 2017 subtitles and on news test data.

Interesting ideas

  • I1. Cheap proxy target for quality: expensive lexical translation probabilities are computed only on a sample, then a fast feature-based regressor extends the estimate to the whole corpus. This pattern of an expensive label on a subset plus a cheap model at scale could be reused for quality estimation in any large crawled corpus.
  • I2. Symmetric, length-normalised scoring: the score takes the worse of the two directional per-token log-probabilities and then rescales it per language pair. This keeps one-sided matches from looking good and makes scores comparable across languages with different statistics.
  • I3. Language-independent surface features: length ratios, cognates, matching punctuation and empty-alignment counts need no language-specific tooling. That suits low-resource pairs, for example Ukrainian with smaller languages.
  • I4. Scores instead of deletion: the release ships a quality score for every pair rather than removing pairs. Each downstream task can then choose its own threshold, a sensible design given that the authors' single fixed threshold did not clearly help NMT.
  • I5. Cleaner in-domain test set: held-out subtitles were limited to one-to-one pairs with over 80% time overlap. Using timing metadata this way is a simple method to build trustworthy evaluation sets from noisy aligned data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cited by

105 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 92 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lison2018opensubtitles2018,
  title = {{OpenSubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora}},
  author = {Lison, Pierre and Tiedemann, Jörg and Kouylekov, Milen},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  doi = {10.63317/4cin6ute7f3b},
  url = {https://doi.org/10.63317/4cin6ute7f3b}
}

References

19 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akbik, A., Kumar, V., and Li, Y. (2016). Towards semi-automatic generation of proposition banks for low-resource languages. In EMNLP, pages 993–998. find
  2. Aziz, W., de Sousa, S. C. M., and Specia, L. (2012). Cross-lingual sentence compression for subtitles. In 16th Annual Conference of the European Association for Machine Translation (EAMT 2012), pages 103–110, Trento, Italy. find
  3. Bolstad, B. M., Irizarry, R. A., Åstrand, M., and Speed, T. P. (2003). A comparison of normalization methods for high density oligonucleotide array data based on variance and bias. Bioinformatics, 19(2):185–193. find
  4. Brown, P. F., Pietra, V. J. D., Pietra, S. A. D., and Mercer, R. L. (1993). The mathematics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2):263–311. find
  5. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C. J., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the A… find
  6. Koehn, P. (2009). Statistical machine translation. Cambridge University Press. find
  7. Lison, P. and Bibauw, S. (2017). Not all dialogues are created equal: Instance weighting for neural conversational models. In Proceedings of the 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL 2017), pages 384–394, Saarbrücken, Germany. ACL. find
  8. Lison, P. and Tiedemann, J. (2016). Opensubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016). find our card
  9. Mikhailov, M. and Cooper, R. (2016). Corpus Linguistics for Translation and Contrastive Studies: a guide for research. Routledge. find
  10. Neubig, G., Nakata, Y., and Mori, S. (2011). Pointwise prediction for robust, adaptable japanese morphological analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL 2011), pages 529–533, Portland, Oregon, USA. find
  11. Och, F. J. and Ney, H. (2000). Giza++: Training of statistical translation models. find
  12. Östling, R., Scherrer, Y., Tiedemann, J., Tang, G., and Nieminen, T. (2017). The Helsinki neural machine translation system. In Proceedings of the Second Conference on Machine Translation, Copenhagen, Denmark, September. Association for Computational Linguistics. find
  13. Paetzold, G. and Specia, L. (2016). Collecting and exploring everyday language for predicting psycholinguistic properties of words. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1669–1679, Osaka, Japan, December. find
  14. The COLING 2016 Organizing Committee. find
  15. Paetzold, G. H. (2016). Lexical Simplification for Non-Native English Speakers. Ph.D. thesis, University of Sheffield. find
  16. Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August. Association for Computational Lin… find
  17. Tiedemann, J. (2008). Synchronizing translated movie subtitles. In Proceedings of the International Conference on Language Resources and Evaluation, LREC 2008, 26 May - 1 June 2008, Marrakech, Morocco. find
  18. Tiedemann, J. (2016). Finding alternative translations in a large corpus of movie subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC-2016). van der Wees, M., Bisazza, A., and Monz, C. (2016). Measuring the effect of conversational aspects on mac… find
  19. Vinyals, O. and Le, Q. (2015). A Neural Conversational Model. CoRR, abs/1506.05869. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/lison-2018-opensubtitles2018-statistical-rescoring-sentence-alignments/