Elamur Research
☰
home / papers / 2020 / ParaCrawl: Web-Scale Acquisition of Parallel Corpora

ParaCrawl: Web-Scale Acquisition of Parallel Corpora

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.417.pdf

Summary

Most openly available parallel corpora come from single, curated sources such as parliament proceedings or subtitles. Large-scale web mining had mostly been done inside big companies, using tools that were never released. ParaCrawl addresses this gap by building openly released bitexts at web scale with open-source software. The authors crawled hundreds of thousands of multilingual websites and ran the results through a pipeline of five stages: fetching the web pages, extracting their text, pairing up documents, matching sentences within those documents, and finally filtering the resulting sentence pairs. All stages are integrated into the modular tool Bitextor. For several stages, they compared competing methods by how much they improved downstream MT quality, and they published the evaluation data as benchmarks. The work centres on official EU languages, with additional effort on Russian and several low-resource languages. The resulting corpora improved state-of-the-art results on benchmarks such as the WMT news translation task. The project matters because it provides reusable data, code and benchmarks for parallel corpus mining. It also argues that MT researchers should report energy consumption alongside BLEU scores.

Key points

  • Release 5 covers 23 EU languages paired with English, plus Russian–English, Dutch–French and Polish–German. Each pair comes in a noisy Raw version and a filtered Clean version.
  • Sentence pairs with a Bicleaner score below 0.7 were removed from Clean but kept in Raw, so users can apply their own filters.
  • French–English is the largest Clean set, with about 51.3M sentence pairs. German–English has the largest Raw set, with about 5.04 billion pairs. Maltese–English is the smallest Clean set, with about 177K pairs.
  • Besides EU languages, the pipeline also targeted Sinhala, Nepali, Tagalog, Swahili and Somali.
  • The evaluation used roughly 7 MWh of energy. This is comparable to a year's use for a high-consuming UK household with electric heating (7.1 MWh, per Ofgem).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

170 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 164 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.417

@inproceedings{banon2020paracrawl,
  title = {{ParaCrawl: Web-Scale Acquisition of Parallel Corpora}},
  author = {Bañón, Marta and Chen, Pinzhen and Haddow, Barry and Heafield, Kenneth and Hoang, Hieu and Esplà-Gomis, Miquel and Forcada, Mikel L. and Kamran, Amir and Kirefu, Faheem and Koehn, Philipp and Rojas, Sergio Ortiz and Sempere, Leopoldo Pla and Ramírez-Sánchez, Gema and Sarrías, Elsa and Strelec, Marek and Thompson, Brian and Waites, William and Wiggins, Dion and Zaragoza, Jaume},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.417},
  url = {https://doi.org/10.18653/v1/2020.acl-main.417}
}

References

67 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ondřej Bojar, Adam Liška, and Zdeněk Žabokrtský. 2010. Evaluating utility of data sources in a large parallel Czech-English corpus CzEng 0.9. In Proceedings of LREC2010. find
  2. Ondřej Bojar, Zdeněk Žabokrtský, Ondřej Dušek, Petra Galuščáková, Martin Majliš, David Mareček, Jiřı́ Maršı́k, Michal Novák, Martin Popel, and Aleš Tamchyna. 2012. The Joy of Parallelism with CzEng 1.0. In Proceedings of LREC2012, Istanbul, Turkey. ELRA, European Language Resources A… find
  3. Peter F. Brown, Jennifer C. Lai, and Robert L. Mercer. 1991. Aligning sentences in parallel corpora. In Proceedings of the 29th Annual Meeting on Association for Computational Linguistics, ACL ’91, pages 169–176, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  4. Christian Buck, Kenneth Heafield, and Bas van Ooyen. 2014. N-gram counts and language models from the common crawl. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reyk-javı́k, Iceland. find
  5. Christian Buck and Philipp Koehn. 2016a. Findings of the wmt 2016 bilingual document alignment shared task. In Proceedings of the First Conference on Machine Translation, pages 554–563, Berlin, Germany. Association for Computational Linguistics. find
  6. Alexandra Antonova and Alexey Misyurev. 2011. Building a web-based parallel corpus and filtering out machine-translated text. In Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web, pages 136–144, Portland, Oregon. Association for Computational Li… find
  7. Christian Buck and Philipp Koehn. 2016b. Quick and reliable document alignment via tf/idf-weighted cosine distance. In Proceedings of the First Conference on Machine Translation, pages 672–678, Berlin, Germany. Association for Computational Linguistics. find
  8. Mikel Artetxe and Holger Schwenk. 2018. Margin-based parallel corpus mining with multilingual sentence embeddings. CoRR, abs/1811.01136. find our card
  9. M. Cettolo, C. Girardi, and M. Federico. 2012. WIT3: Web inventory of transcribed and translated talks. In Proceedings of th 16th International Conference of the European Association for Machine Translation (EAMT), pages 261–268. find our card
  10. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 597–604, Ann Arbor, Michigan. Association for Computational Linguistics. find our card
  11. Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. 2019. Low-resource corpus filtering using multilingual sentence embeddings. In Proceedings of the Fourth Conference on Machine Translation (WMT). find
  12. Gabriel Bernier-Colborne and Chi-kiu Lo. 2019. NRC parallel corpus filtering system for WMT 2019. In Proceedings of the Fourth Conference on Machine Translation (WMT). find
  13. Ondřej Bojar, Ondřej Dušek, Tom Kocmi, Jindřich Libovický, Michal Novák, Martin Popel, Roman Sudarikov, and Dušan Variš. 2016. CzEng 1.6: Enlarged Czech-English Parallel Corpus with Processing Tools Dockered. In Text, Speech, and Dialogue: 19th International Conference, TSD 2016, number 9924… find
  14. Aswarth Abhilash Dara and Yiu-Chang Lin. 2016. Yoda system for wmt16 shared task: Bilingual document alignment. In Proceedings of the First Conference on Machine Translation. find
  15. Grant Erdmann and Jeremy Gwinnup. 2019. Quality and coverage: The afrl submission to the wmt19 parallel corpus filtering for low-resource conditions task. In Proceedings of the Fourth Conference on Machine Translation (WMT). find
  16. Miquel Esplà-Gomis, Mikel Forcada, Sergio Ortiz Rojas, and Jorge Ferrández-Tordera. 2016. Bitextor’s participation in wmt’16: shared task on document alignment. In Proceedings of the First Conference on Machine Translation, pages 685–691, Berlin, Germany. Association for Computational Linguistics. find
  17. Ken’ichi Fukushima, Kenjiro Taura, and Takashi Chikayama. 2006. A fast and accurate method for detecting English-Japanese parallel texts. In Proceedings of the Workshop on Multilingual Language Resources and Interoperability, pages 60–67, Sydney, Australia. Association for Computational Linguistics. find
  18. William A Gale and Kenneth W Church. 1993. A program for aligning sentences in bilingual corpora. Computational linguistics, 19(1):75–102. find our card
  19. Ulrich Germann. 2016. Bilingual document alignment with latent semantic indexing. In Proceedings of the First Conference on Machine Translation, pages 692–696, Berlin, Germany. Association for Computational Linguistics. find
  20. Luı́s Gomes and Gabriel Pereira Lopes. 2016. First steps towards coverage-based sentence alignment. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 2228–2231, Portorož, Slovenia. European Language Resources Association (ELRA). find
  21. Jesús González-Rubio. 2019. Webinterpret submission to the wmt2019 shared task on parallel corpus filtering. In Proceedings of the Fourth Conference on Machine Translation (WMT). find
  22. Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernand ez Abrego, Keith Stevens, Noah Constant, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Effective parallel corpus mining using bilingual sentence embeddings. In Proceedings of the Third Conference on Machine Translati… find
  23. Mandy Guo, Yinfei Yang, Keith Stevens, Daniel Cer, Heming Ge, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Hierarchical document encoder for parallel corpus mining. In Proceedings of the Fourth Conference on Machine Translation, pages 64–72, Florence, Italy. Association for Computational Li… find
  24. J. Edward Hu, Rachel Rudinger, Matt Post, and Benjamin Van Durme. 2019. Parabank: Monolingual bitext generation and sentential paraphrasing via lexically-constrained neural machine translation. In Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19). find
  25. International Organization for Standardization ISO. 2009. ISO 28500:2009 information and documentation-WARC file format. find
  26. Marcin Junczys-Dowmunt. 2018. Dual conditional cross-entropy filtering of noisy parallel corpora. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational Linguistics. find
  27. Heiki-Jaan Kaalep and Kaarel Veskis. 2007. Comparing parallel corpora and evaluating their quality. In Proceedings of the MT Summit XI. find
  28. Huda Khayrallah and Philipp Koehn. 2018. On the impact of various types of noise on neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 74–83. Association for Computational Linguistics. find our card
  29. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the Tenth Machine Translation Summit (MT Summit X), Phuket, Thailand. find
  30. Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan Pino. 2019. Findings of the WMT 2019 shared task on parallel corpus filtering for low-resource conditions. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 54–72, Florence, Italy.… find
  31. Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L. Forcada. 2018. Findings of the wmt 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 726–739, Belgium, Brussels. Association for Computational Linguist… find
  32. Murathan Kurfalı and Robert Östling. 2019. Noisy parallel corpus filtering through projected word embeddings. In Proceedings of the Fourth Conference on Machine Translation (WMT). find
  33. Thanh Le, Hoa Trong Vu, Jonathan Oberländer, and Ondřej Bojar. 2016. Using term position similarity and language modeling for bilingual document alignment. In Proceedings of the First Conference on Machine Translation, pages 710–716, Berlin, Germany. Association for Computational Linguistics. find
  34. Bo Li and Juan Liu. 2008. Mining Chinese-English parallel corpora from the web. In Proceedings of the 3rd International Joint Conference on Natural Language Processing (IJCNLP). find
  35. Pierre Lison and Jörg Tiedemann. 2016. Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Language Resources and Evaluation Conference (LREC). find
  36. Pintu Lohar, Haithem Afli, Chao-Hong Liu, and Andy Way. 2016. The adapt bilingual document alignment system at wmt16. In Proceedings of the First Conference on Machine Translation, pages 717– 723, Berlin, Germany. Association for Computational Linguistics. find
  37. Lieve Macken, Julia Trushkina, and Lidia Rura. 2007. Dutch parallel corpus: MT corpus and translator’s aid. In Proceedings of the MT Summit XI. find
  38. Joel Martin, Howard Johnson, Benoit Farley, and Anna Maclachlan. 2003. Aligning and using an English-Inuktitut parallel corpus. In HLT-NAACL 2003 Workshop: Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, Edmonton, Alberta, Canada. Association for Computational Linguist… find
  39. Robert C Moore. 2002. Fast and accurate sentence alignment of bilingual corpora. In Conference of the Association for Machine Translation in the Americas, pages 135–144. Springer. find
  40. Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving machine translation performance by exploiting non-parallel corpora. Computational Linguistics, 31(4). find our card
  41. Vassilis Papavassiliou, Prokopis Prokopidis, and Stelios Piperidis. 2016. The ilsp/arc submission to the wmt 2016 bilingual document alignment shared task. In Proceedings of the First Conference on Machine Translation, pages 733–739, Berlin, Germany. Association for Computational Linguistics. find
  42. Zuzanna Parcheta, Germán Sanchis-Trilles, and Francisco Casacuberta. 2019. Filtering of noisy parallel corpora based on hypothesis generation. In Proceedings of the Fourth Conference on Machine Translation (WMT). find
  43. Alexandre Rafalovitch and Robert Dale. 2009. United Nations General Assembly resolutions: A six-language parallel corpus. In Proceedings of the Twelfth Machine Translation Summit (MT Summit XII). International Association for Machine Translation. find
  44. Spencer Rarrick, Chris Quirk, and Will Lewis. 2011. MT detection in web-scraped parallel corpora. In Proceedings of the 13th Machine Translation Summit (MT Summit XIII), pages 422–430. International Association for Machine Translation. find
  45. Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019. Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. CoRR, abs/1907.05791. find our card
  46. Sukanta Sen, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Parallel corpus filtering based on fuzzy string matching. In Proceedings of the Fourth Conference on Machine Translation (WMT). find
  47. Rico Sennrich and Martin Volk. 2010. MT-based sentence alignment for OCR-generated parallel texts. In The Ninth Conference of the Association for Machine Translation in the Americas (AMTA 2010). find
  48. Rico Sennrich and Martin Volk. 2011. Iterative, MT-based sentence alignment of parallel texts. In Proceedings of the 18th Nordic Conference of Computational Linguistics (NODALIDA 2011), pages 175– 182. find
  49. Lei Shi, Cheng Niu, Ming Zhou, and Jianfeng Gao. 2006. A dom tree alignment model for mining parallel data from the web. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 489–496. Assoc… find
  50. Felipe Soares and Marta R. Costa-jussà. 2019. Unsupervised corpus filtering and mining. In Proceedings of the Fourth Conference on Machine Translation (WMT). find
  51. Wolfgang Täger. 2011. The sentence-aligned european patent corpus. In Proceedings of the 15th International Conference of the European Association for Machine Translation (EAMT), pages 177–184. find
  52. Philip Resnik. 1999. Mining the web for bilingual text. In Proceedings of the 37th Annual Meeting of the Association of Computational Linguistics (ACL). find
  53. Brian Thompson and Philipp Koehn. 2019. Vecalign: Improved sentence alignment in linear time and space. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Hong Kong. Association for Computational Linguistics. find
  54. Philip Resnik and Noah A Smith. 2003. The web as a parallel corpus. Computational Linguistics, 29(3):349–380. find our card
  55. Jörg Tiedemann. 2011. Bitext Alignment. Synthesis Lectures on Human Language Technologies. Morgan & Claypool, San Rafael, CA. find
  56. Nick Rossenbach, Jan Rosendahl, Yunsu Kim, Miguel Graça, Aman Gokrani, and Hermann Ney. 2018. The RWTH Aachen University filtering system for the WMT 2018 parallel corpus filtering task. In Proceedings of the Third Conference on Machine Translation, Belgium, Brussels. Association for Computational… find
  57. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA). ACL Anthology Identifier: L12-1246. find
  58. Vı́ctor M. Sánchez-Cartagena, Marta Bañón, Sergio Ortiz Rojas, and Gema Ramı́rez. 2018. Prompsit’s submission to wmt 2018 parallel corpus filtering shared task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 955–962, Belgium, Brussels. Association for Com… find
  59. Masao Uchiyama and Hitoshi Isahara. 2007. A Japanese-English patent parallel corpus. In Proceedings of the MT Summit XI. find
  60. Jakob Uszkoreit, Jay Ponte, Ashok Popat, and Moshe Dubiner. 2010. Large scale parallel document mining for machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1101–1109, Beijing, China. find
  61. Coling 2010 Organizing Committee. find
  62. Masao Utiyama and Hitoshi Isahara. 2003. Reliable measures for aligning Japanese-English news articles and sentences. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 72–79. find
  63. Dániel Varga, Péter Halaácsy, András Kornai, Vok-tor Nagy, László Németh, and Viktor Trón. 2005. Parallel corpora for medium density languages. In Proceedings of the RANLP 2005 Conference, pages 590–596. find
  64. Ashish Venugopal, Jakob Uszkoreit, David Talbot, Franz Och, and Juri Ganitkevitch. 2011. Watermarking the outputs of structured prediction with an application in statistical machine translation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1363–137… find
  65. Hainan Xu and Philipp Koehn. 2017. Zipporah: a fast and scalable data cleaning system for noisy web-crawled parallel corpora. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2935–2940. Association for Computational Linguistics. find
  66. David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of the First International Conference on Human Language Technology Research. find
  67. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2015. The united nations parallel corpus v1.0. In International Conference on Language Resources and Evaluation (LREC). find our card