Elamur Research⌕
☰
home / papers / 2013 / Dirt Cheap Web-Scale Parallel Text from the Common Crawl

Dirt Cheap Web-Scale Parallel Text from the Common Crawl

Summary

Statistical machine translation depends on parallel text. Curated corpora cover only a few language pairs, and most of them are government or news text. Large web crawls could supply more, but only big companies have been able to crawl and process the web at that scale. The authors avoid crawling altogether by mining the public Common Crawl directly inside Amazon's cloud. Their open-source pipeline extends the STRAND algorithm:

  • It pairs pages whose URLs differ only by a language code or language name, using a Map-Reduce job.
  • It aligns the linearized HTML of each pair and splits the text with NLTK's Punkt.
  • It aligns sentences with Gale and Church (1993) and removes repeated, boilerplate-like segments.

Processing 32.3 TB of the crawl took less than a day and cost roughly $500. The output includes parallel data for dozens of languages paired with English, some of which had little curated data before. Adding this lightly filtered data improved news translation for five language pairs. On open-domain test sets the gains reached about 5 BLEU. This shows that web-scale parallel data is now affordable for ordinary research groups.

Key points

  • The run covered the 2009–2010 Common Crawl (32.3 TB). Compute cost about $400 and output storage/transfer about $100. Downloading the full 81 TB crawl instead would have cost over $8000.

  • Yields with English varied widely:

    • High-resource: French 10.2M segments, German 7.50M, Spanish 5.67M, Russian 3.58M.
    • Low-resource: Pashto 28.0K, Kannada 34.5K, Somali 52.6K.

  • Recall was checked against URLs from the French-English Gigaword corpus. The URL heuristics could have matched 45% of those pairs, but only 3.6% appeared in the output. Allowing single-letter codes ("f"/"e") would raise potential coverage to 74%, but these were left out because they produced too many false positives.

  • The authors trained a maximum entropy classifier for structural filtering on 101 annotated Spanish-English document pairs. It did no better than simply labelling every pair parallel, because 80% of the sample was parallel. They therefore dropped the classifier.

  • Spanish-English BLEU, Europarl baseline vs. Europarl + Wiki + Web data:

    Test setEuroparl+ Wiki + Web
    WMT1027.2128.74
    Tatoeba36.1341.12
    Fisher46.3252.23

    Code is published at github.com/jrs026/CommonCrawlMiner.

Interesting ideas

  • I1. URL masking as a Map-Reduce key. Replacing the language marker in a URL with a wildcard gives a language-independent key, so the reducer collects possible translations of the same page together. The approach needs only a list of language codes and runs in roughly linear time, which suits languages with no seed resources.
  • I2. Processing data where it is stored. Running the job inside the cloud that hosts the dataset avoids heavy transfer fees. Any project working with large public corpora can cut costs the same way.
  • I3. Checking a learned component against a trivial baseline. The filtering classifier was compared with a label-everything-parallel baseline, which revealed that it added nothing on the imbalanced sample. Making this comparison before keeping a component helps avoid unnecessary complexity.
  • I4. Building a blind test set. The authors had Fisher Spanish speech transcripts translated through Mechanical Turk, so the references could not exist anywhere on the web. This rules out contamination when evaluating systems trained on web data.
  • I5. Using crude results to bootstrap better methods. Even a recall-poor first pass gives a lower bound on what is possible. It also supplies seed parallel data for stronger methods, such as mining comparable Wikipedia text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation · Natural language processing

Topics: Machine translation

Cited by

63 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 55 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{smith2013dirt,
  title = {{Dirt Cheap Web-Scale Parallel Text from the Common Crawl}},
  author = {Smith, Jason R. and Saint-Amand, Herve and Plamada, Magdalena and Koehn, Philipp and Callison-Burch, Chris and Lopez, Adam},
  booktitle = {Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2013},
  doi = {10.5167/uzh-80038},
  url = {https://doi.org/10.5167/uzh-80038}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993–1022, March. find our card
  2. Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar F. Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, WMT ’10, pages 17–53. Association for Computational Linguistics. find our card
  3. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar F. Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, WMT ’11, pages 22–64. Association for Computational Linguistics. find our card
  4. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 workshop on statistical machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10–51, Montréal, Canada, June. Association for Computational Linguistics. find our card
  5. Jiang Chen and Jian-Yun Nie. 2000. Parallel web text mining for cross-language ir. In IN IN PROC. OF RIAO, pages 62–77. find
  6. J. Dean and S. Ghemawat. 2004. Mapreduce: simplified data processing on large clusters. In Proceedings of the 6th conference on Symposium on Opeart-ing Systems Design & Implementation-Volume 6, pages 10–10. USENIX Association. find
  7. William A. Gale and Kenneth W. Church. 1993. A program for aligning sentences in bilingual corpora. Comput. Linguist., 19:75–102, March. ↗ our card
  8. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, ACL ’07, pages 177–180. Association for Computational Linguistics. find
  9. P. Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit, volume 5. find
  10. Edward Loper and Steven Bird. 2002. Nltk: the natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics - Volume 1, ETMTNLP ’02, pages 63–70. Association for Computational Linguistics. ↗ our card
  11. Adam Lopez, Matt Post, and Chris Callison-Burch. 2013. Parallel speech, transcription, and translation: The Fisher and Callhome Spanish-English speech translation corpora. Technical Report 11, Johns Hopkins University Human Language Technology Center of Excellence. find
  12. Marco Lui and Timothy Baldwin. 2012. langid.py: an off-the-shelf language identification tool. In Proceedings of the ACL 2012 System Demonstrations, ACL ’12, pages 25–30. Association for Computational Linguistics. find our card
  13. Andrew Kachites McCallum. 2002. Mallet: A machine learning for language toolkit. http://mallet.cs.umass.edu. ↗
  14. Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving Machine Translation Performance by Exploiting Non-Parallel Corpora. Comput. Linguist., 31:477–504, December. find our card
  15. Jian-Yun Nie, Michel Simard, Pierre Isabelle, and Richard Durand. 1999. Cross-language information retrieval based on parallel texts and automatic mining of parallel texts from the web. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, SIGIR ’99, pages 74–81, New York, NY, USA. ACM. find
  16. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In acl, pages 160– 167, Sapporo, Japan. find
  17. P. Resnik and N. A Smith. 2003. The web as a parallel corpus. Computational Linguistics, 29(3):349–380. ↗ our card
  18. Philip Resnik. 1999. Mining the web for bilingual text. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, ACL ’99, pages 527–534. Association for Computational Linguistics. find
  19. Jason R. Smith, Chris Quirk, and Kristina Toutanova. 2010. Extracting Parallel Sentences from Comparable Corpora using Document Level Alignment. In NAACL 2010. find our card
  20. Jörg Tiedemann. 2009. News from OPUS - A collection of multilingual parallel corpora with tools and interfaces. In N. Nicolov, K. Bontcheva, G. Angelova, and R. Mitkov, editors, Recent Advances in Natural Language Processing, volume V, pages 237–248. John Benjamins, Amsterdam/Philadelphia, Borovets, Bulgaria. ↗
  21. Ferhan Ture and Jimmy Lin. 2012. Why not grab a free lunch? mining large corpora for parallel sentences to improve translation modeling. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 626–630, Montréal, Canada, June. Association for Computational Linguistics. find
  22. Jakob Uszkoreit, Jay M. Ponte, Ashok C. Popat, and Moshe Dubiner. 2010. Large scale parallel document mining for machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics, COLING ’10, pages 1101– 1109. Association for Computational Linguistics. find
  23. Ashish Venugopal, Jakob Uszkoreit, David Talbot, Franz J. Och, and Juri Ganitkevitch. 2011. Watermarking the outputs of structured prediction with an application in statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’11, pages 1363–1372. Association for Computational Linguistics. find
  24. Omar F. Zaidan and Chris Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proc. of ACL. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/smith-2013-dirt-cheap-web-scale-parallel/