ParaCrawl: Web-scale parallel corpora for the languages of the EU

Summary

Machine translation for many European languages is held back by a lack of parallel training data. This extended abstract describes ParaCrawl and its successor Paracrawl-2, two Connecting Europe Facility actions that mine the web for translated text in EU languages. The completed first action paired English with each of the other 23 official EU languages. The ongoing second action adds new combinations, including Spanish with Catalan, Basque and Galician, and the two Norwegian written standards.

A consortium of five university and industry partners releases the corpora under the CC0 license. It also maintains the open-source Bitextor pipeline, which handles every step from downloading HTML pages to producing a cleaned, deduplicated bitext, and includes the Bicleaner noise filter. Experts checked samples of version 3, and the problems they found were fixed before version 4 was released. Adding version 4 to WMT 2018 baselines raised BLEU for almost every pair tested. The project matters because it supplies both large free datasets and scalable, reusable tools for building more of them.

Key points

ParaCrawl version 4 covers English paired with 23 EU languages. Sizes range from about 195K segment pairs (Maltese) to about 31.4M (French), with Spanish at about 22.0M and German at about 16.3M. The data is available on paracrawl.eu and ELRC-share.
Quality checks: language experts reviewed 2,000-sentence random samples per pair in version 3. Adding version 4 data to WMT 2018 baselines improved BLEU in 11 of 12 language pairs.
Bitextor 7 is a five-stage pipeline:
1. crawling
2. preprocessing and normalization
3. document alignment
4. segment alignment
5. filtering, deduplication and formatting
It is modular, scales to clusters (SLURM, PBS Pro) and cloud platforms (Azure), and its components can also be used on their own.
Bicleaner, the integrated filtering tool, was among the top systems in the WMT 2018 parallel corpus cleaning shared task.
Planned work includes:
- adding Icelandic, Norwegian, Basque, Catalan/Valencian and Galician
- handling non-HTML formats (PDF, DOCX, ODT)
- domain identification
- mining the Internet Archive
- repairing noisy segments rather than discarding them
A further action, Paracrawl-3 (2018-EU-IA-0063), has already been funded.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

61 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 54 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{espla2019paracrawl,
  title = {{ParaCrawl: Web-scale parallel corpora for the languages of the EU}},
  author = {Esplà, Miquel and Forcada, Mikel and Ramírez-Sánchez, Gema and Hoang, Hieu},
  booktitle = {Proceedings of Machine Translation Summit XVII: Translator, Project and User Tracks},
  year = {2019},
  url = {https://aclanthology.org/W19-6721/}
}

References

1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Sánchez-Cartagena, Vı́ctor M., Marta Bañón, Sergio Ortiz-Rojas, and Gema Ramı́rez-Sánchez. 2018. Prompsit’s submission to the WMT 2018 parallel corpus filtering shared task. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 968–975, Brussels, Belg… find

This page in Markdown → https://md.research.elamur.ai/free/papers/espla-2019-paracrawl-web-scale-parallel-corpora/

📄 ParaCrawl: Web-scale parallel corpora for the languages of the EU