Intelligent Selection of Language Model Training Data

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P10-2041.pdf

Summary

Language models for tasks like machine translation benefit from more training data only when that data resembles the target domain. However, in-domain text is usually scarce, and large general corpora are only partly relevant. Moore and Lewis propose filtering a general corpus sentence by sentence before building an auxiliary language model. Each candidate sentence receives a score equal to its per-word cross-entropy under an in-domain model minus its cross-entropy under a model built from randomly drawn general-corpus text. Sentences scoring below a threshold, tuned on held-out in-domain text, are kept. Normalizing by length prevents a bias toward very short sentences. The authors use the English side of Europarl v5 as the domain and LDC English Gigaword Third Edition as the general pool. They compare their method against random sampling, in-domain perplexity filtering (Lin et al.; Gao et al.) and Klakow's unigram method. The cross-entropy difference approach reaches the lowest test perplexity while keeping under 7% of Gigaword. This matters because filtering needs far less memory than training on everything, although BLEU gains remain to be shown.

Key points

• Data:
  • In-domain training set: Europarl 1999–2008, with about 1.65M sentences and 48.2M tokens.
  • Test set: the first 2,000 sentences from January 2009.
  • Gigaword pool: about 133M sentences and 3.45B tokens.
  • The scoring model for Gigaword was trained on a random sample of comparable size (about 1.87M sentences).
• Setup: Both scoring models are 4-gram models with absolute discounting (discount 0.7). Their vocabulary is limited to tokens seen at least twice in Europarl training. Eight score cutoffs were evaluated.
• Perplexity results:
  • Full Gigaword: 135.
  • In-domain cross-entropy filtering: 124, using 36% of the data.
  • Klakow's method: 111, using 21%.
  • Cross-entropy difference: 101, using less than 7%.
• Why length normalization is needed: Raw log-probability differences correlate strongly with sentence length (r = −0.92). Cross-entropy differences are nearly independent of length (r = 0.04).
• Vocabulary-controlled check: After equalizing out-of-vocabulary (OOV) handling, perplexities were 124.8, 110.8 and 101.9. The ranking of methods is unchanged.