Elamur Research
☰
home / papers / 2020 / CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs

CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.480.pdf

Summary

Finding web pages that translate one another is valuable for machine translation and other cross-lingual NLP work. Labeling such pairs by hand is costly, because the number of candidate pairs grows quadratically, and it is especially difficult for low-resource languages. El-Kishky et al. use weak supervision from URLs instead. They deduplicate 68 Common Crawl snapshots, tag each page's language with FastText, and pair URLs that become identical once language markers (such as "en-gb" or "lang=fr") are removed. A pair is kept only if the removed marker agrees with the detected language of the page.

The resulting dataset, CCAligned, is a URL-level collection of document pairs. Human annotators judged it highly precise, though the URL rules sacrifice recall. The authors also provide a benchmark subset and baselines that align documents using embeddings of their content. Sentences mined from the English-paired documents train NMT models whose quality is comparable to models trained on ParaCrawl. CCAligned therefore greatly extends multilingual coverage, particularly for low-resource and non-European language pairs.

Key points

  • Scale: The source was 169.4B raw documents from 68 Common Crawl snapshots (2013–2020). Deduplication reduced this to about 29.6B documents from 107.8M domains, an 83% cut.
  • Output: Over 392M aligned document pairs across 8144 language pairs and 138 languages. About 100M pairs include English (137 English pairs) and 292M do not.
  • Human evaluation: Annotators checked 180 URL pairs, 30 each for German, Chinese, Arabic, Romanian, Estonian and Burmese. Each pair was rated by three people.
    • Average majority-vote precision was 84.4%.
    • Precision adjusted for experimental error was 94.5%.
    • Krippendorff's α was 0.70.
  • Benchmark set: 121K documents from 450 web domains, containing 17.5K URL-aligned pairs, for testing content-based alignment methods.
  • Downstream MT:
    • LASER with margin-based filtering mined over 1B unique parallel sentences.
    • fairseq transformer models were trained on these sentences and tested on TED talk test sets.
    • BLEU was roughly on par with ParaCrawl v6 and higher than WikiMatrix.
    • The URL lists and code are released at statmt.org/cc-aligned.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

131 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 124 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.480

@inproceedings{elkishky2020ccaligned,
  title = {{CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs}},
  author = {El-Kishky, Ahmed and Chaudhary, Vishrav and Guzmán, Francisco and Koehn, Philipp},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.480},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.480}
}

References

37 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sadaf Abdul-Rauf and Holger Schwenk. 2009. On the use of comparable corpora to improve smt performance. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, pages 16–23. Association for Computational Linguistics. find
  2. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. find our card
  3. Christian Buck, Kenneth Heafield, and Bas van Ooyen. 2014. N-gram counts and language models from the common crawl. In Proceedings of the Language Resources and Evaluation Conference, Reykjavík, Iceland. find
  4. Christian Buck and Philipp Koehn. 2016a. Findings of the wmt 2016 bilingual document alignment shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 554–563. find
  5. Christian Buck and Philipp Koehn. 2016b. Quick and reliable document alignment via tf/idf-weighted cosine distance. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 672–678. find
  6. Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. 2019. Low-resource corpus filtering using multilingual sentence embeddings. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 263–268, Florence, Italy. Assoc… find
  7. Jiang Chen and Jian-Yun Nie. 2000. Parallel web text mining for cross-language ir. In Content-Based Multimedia Information Access-Volume 1, pages 62–77. Le Centre de Haute Etudes Internationales d’Informatique Documentaire. find
  8. Aswarth Abhilash Dara and Yiu-Chang Lin. 2016. Yoda system for wmt16 shared task: Bilingual document alignment. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 679–684. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  10. Thi-Ngoc-Diep Do, Viet-Bac Le, Brigitte Bigi, Laurent Besacier, and Eric Castelli. 2009. Mining a comparable text corpus for a vietnamese-french statistical machine translation system. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 165–172. Association for Computatio… find
  11. Miquel Espla-Gomis and Mikel Forcada. 2010. Combining content-based and url-based heuristics to harvest aligned bitexts from multilingual sites with bitextor. The Prague Bulletin of Mathematical Linguistics, 93(2010):77–86. find
  12. Miquel Esplà-Gomis, Mikel L Forcada, Gema Ramírez-Sánchez, and Hieu Hoang. 2019. Paracrawl: Web-scale parallel corpora for the languages of the eu. In Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks, pages 118–119. find
  13. Luís Gomes and Gabriel Pereira Lopes. 2016. First steps towards coverage-based document alignment. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 697–702. find
  14. Mandy Guo, Yinfei Yang, Keith Stevens, Daniel Cer, Heming Ge, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Hierarchical document encoder for parallel corpus mining. In Proceedings of the Fourth Conference on Machine Translation, pages 64–72, Florence, Italy. Association for Computational Li… find
  15. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala– English. In Proceedings of the 2019 Conference on Empirical Metho… find our card
  16. Armand Joulin, Edouard Grave, and Piotr Bojanowski Tomas Mikolov. 2017. Bag of tricks for efficient text classification. find
  17. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit, volume 5, pages 79–86. find
  18. Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan Pino. 2019. Findings of the WMT 2019 shared task on parallel corpus filtering for low-resource conditions. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 54–72, Florence, Italy.… find
  19. Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L. Forcada. 2018. Findings of the WMT 2018 shared task on parallel corpus filtering. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 726–739, Belgium, Brussels. Association for Computational Linguist… find
  20. Klaus Krippendorff. 2011. Computing Krippendorff’s alpha-reliability. Technical report, University of Pennsylvania, Annenberg School for Communication. find
  21. Alexandre Rafalovitch, Robert Dale, et al. 2009. United nations general assembly resolutions: A six-language parallel corpus. In Proceedings of Machine Translation Summit XII. find
  22. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71. find our card
  23. Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, volume 242, pages 133–142. Piscataway, NJ. find
  24. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  25. Philip Resnik. 1999. Mining the web for bilingual text. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, pages 527–534. Association for Computational Linguistics. find
  26. James Munkres. 1957. Algorithms for the assignment and transportation problems. Journal of the society for industrial and applied mathematics, 5(1):32–38. find
  27. Philip Resnik and Noah A Smith. 2003. The web as a parallel corpus. Computational Linguistics, 29(3):349–380. find our card
  28. Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving machine translation performance by exploiting non-parallel corpora. Computational Linguistics, 31(4):477–504. find our card
  29. Holger Schwenk. 2018. Filtering and mining parallel data in a joint multilingual space. arXiv preprint arXiv:1805.09822. find
  30. Dragos Stefan Munteanu and Daniel Marcu. 2006. Extracting parallel sub-sentential fragments from non-parallel corpora. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 81–88. Associati… find
  31. Jian-Yun Nie, Michel Simard, Pierre Isabelle, and Richard Durand. 1999. Cross-language information retrieval based on parallel texts and automatic mining of parallel texts from the web. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information re… find
  32. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr… find our card
  33. Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua… find our card
  34. Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019. Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. arXiv preprint arXiv:1907.05791. find our card
  35. Jason R Smith, Herve Saint-Amand, Magdalena Plamada, Philipp Koehn, Chris Callison-Burch, and Adam Lopez. 2013. Dirt cheap web-scale parallel text from the common crawl. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1374–138… find
  36. Raghavendra Udupa, K Saravanan, A Kumaran, and Jagadeesh Jagarlamudi. 2009. Mint: A method for effective and scalable mining of named entity transliterations from large comparable corpora. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics,… find
  37. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations parallel corpus v1. 0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 3530–3534. find our card