🧑‍🔬 Guillaume Wenzek

Guillaume Wenzek works on building large text resources for natural language processing out of web crawl data. Their papers here turn raw Common Crawl snapshots into material for training multilingual models. Two concerns run through them: keeping text quality high as corpora grow, and serving languages that have little curated text, such as those with a small Wikipedia. The methods are large-scale processing pipelines and multilingual sentence embeddings applied across dozens of languages.

The most cited of these papers is CCNet (2020). It describes an open-source pipeline that turns the plain-text dumps of Common Crawl into big monolingual corpora while keeping each document whole rather than breaking it into fragments. The companion paper, CCMatrix (2019), takes margin-scored bitext mining with LASER embeddings, an approach first used on Wikipedia, and extends it to the web. Its source is ten CCNet-filtered Common Crawl snapshots, amounting to 32.7 billion distinct sentences across 38 languages. Most mining systems first pair up documents and only then search for matching sentences within them. CCMatrix drops that stage and compares each source sentence with each target sentence in a single shared embedding space. The goal is to provide parallel text for pairs of languages where neither side is English, and for subjects beyond the formal, political material found in parliamentary and UN proceedings.

Taken together, the two papers, from 2019 and 2020, form one line of work: first cleaned monolingual web text, then parallel sentences mined from it. Wenzek's current affiliation is Meta in Menlo Park, and earlier bylines list Meta offices in Tel Aviv and London. Co-authors on papers here include Armand Joulin and Edouard Grave, with two papers each. Others are Holger Schwenk, Alexis Conneau, Vishrav Chaudhary and Francisco Guzmán.

Guillaume Wenzek has 2 papers in our library, in 2019–2020; often with Armand Joulin, Edouard Grave, Alexis Conneau.