📄 Discriminative Corpus Weight Estimation for Machine Translation

Summary

Parallel corpora for SMT mix collections that differ in quality and in how well they match the target domain. For example, there may be abundant UN and newswire text but very little broadcast conversation. Matsoukas, Rosti and Zhang (BBN) learn a weight for every training sentence pair. These weights scale the phrase and lexical counts when translation rules are extracted and their probabilities are estimated. The bitext contains millions of sentences, so the authors do not estimate the weights one by one. Instead, each weight is produced by a perceptron from sentence-level features. The perceptron's parameters are tuned by gradient-based minimisation of expected TER, computed over N-best lists from a phrase-based decoder on a held-out tuning set. In the experiments the only features were collection and genre identifiers. On Arabic-English newswire, web and audio (BN/BC) data, the learned weights improved TER, BLEU and METEOR for the phrase-based system. They also helped a strong hierarchical string-to-dependency-tree system without retuning. The method matters because it adapts the domain and suppresses noisy data automatically, without hand-labelling which data is in-domain and without discarding any data.

Key points

• Weighted estimation. Each sentence weight lies in (0, 1) and multiplies that sentence's phrase-pair and phrase counts in the relative-frequency estimate of P(s|t). The set of extracted rules stays the same; only the probabilities change.
• Features used. Every sentence was encoded as a 41-dimensional binary vector for its collection/genre combination, which a perceptron mapped to a single weight. Alignments came from GIZA++ and were shared by both MT systems.
• Phrase-based results (absolute TER gains).
  • Newswire: 0.8 on both Tune and Test.
  • Web: 0.9 on Tune and 0.5 on Test.
  • Audio Test: 0.5 on BN and 1.4 on BC.
  • BLEU and METEOR also improved in every condition.
• Hierarchical results. The weights were estimated with the phrase-based system and then reused in the hierarchical system. That system scores 52.20 case-sensitive BLEU on the NIST MT08 newswire set, comparable to the second-best unconstrained MT08 entry. On Test the reused weights gave:
  • BLEU +0.8 on newswire.
  • TERBLEU reductions of 0.3 on web, 0.7 on BN and 0.75 on BC.
• Stated limitations. The method cannot add new translation rules. It may overfit if the mapping function has many parameters or if the tuning set does not resemble the test set.

Interesting ideas

• I1. Parameterise weights through features. Predicting per-example weights from a small feature-based function keeps the optimisation tractable for millions of examples. The same trick could weight mixed-source training data for other tasks, such as a parser trained on several treebanks.
• I2. Expected metric over N-best lists. Using expected TER over a fixed N-best list gives a smooth, differentiable objective and avoids re-decoding at every iteration. This is a general way to tune upstream data or model choices against a downstream metric that is not differentiable.
• I3. Soft down-weighting instead of filtering. No data is thrown away, but parts judged harmful can still be pushed close to zero weight. This is a safer default than hard filtering when it is unclear which sources help.
• I4. Weights transfer across architectures. Weights learned with a cheap, fast system improved a much stronger one. This suggests running expensive data-weighting searches on a lightweight proxy model.
• I5. Finer granularity as a next step. The authors propose weighting individual alignment links, because many sentence pairs are only partly correct. The same idea applies to any noisy parallel or annotated data, where errors are often local rather than affecting a whole example.