📄 Pronunciation Modeling for Improved Spelling Correction

Summary

The paper targets spelling correction for errors that come from not knowing how a word is spelled, where the writer produces a form that sounds like the intended word (e.g., latecks for latex). The strong letter-based noisy channel model of Brill and Moore (2000) still fails on many such cases. Toutanova and Moore keep the Brill and Moore learning algorithm but apply it twice on the same misspelling/correct-word pairs. The first application gives a letter-level error model (LTR). The second gives a phone-sequence error model (PH), trained on pronunciations of correct words and automatically predicted pronunciations of misspellings. Those predictions come from an improved N-gram letter-to-phone converter learned from the Microsoft Speech dictionary. A log-linear combination (CMB) rescores the merged candidate lists of both models. On roughly 10,000 English misspelling pairs, CMB reduces 1-best errors by about 24% relative to LTR, and the reduction approaches 47% at 4-best. This shows that explicitly modeling sound-alike errors complements letter-edit models.

Key points

• Accuracy on the test set:
  • LTR alone: 1-best 94.21%, 4-best 99.06%.
  • PH alone: 86.36%, 4-best 96.63%.
  • CMB: 95.58%, 4-best 99.50%.
  • Relative error reduction of CMB over LTR: 23.8% (1-best), 39.6% (2-best), 40% (3-best), 46.8% (4-best).
• Data: about 10,000 misspelling pairs, split into 7,385 training and 1,812 test pairs. Pairs whose correct word was absent from the pronunciation dictionary were removed.
• The letter-to-phone model extends Fisher's (1999) N-gram rule model with five extensions, which together cut its error rate by roughly 20%. The extensions include:
  • interpolating all applicable context rules;
  • marking word-initial and word-final positions explicitly;
  • rescoring N-best pronunciations with a phone trigram LM;
  • rescoring with a vowel 4-gram LM, which had not been proposed before.
• Pronunciation resources:
  • NETtalk (14,876 train / 4,964 test words), which is phone-aligned and was used to check the automatic aligner.
  • Microsoft Speech dictionary (106,650 / 30,003), used to train the final model.
• Letter–phone alignment uses hard EM (Viterbi training). Each letter may map to zero, one, or two phones.

Interesting ideas

• I1. Same learner, different representation. Training an identical error-model algorithm once on characters and once on phones yields two complementary views at almost no extra design cost. The same trick could pair surface forms with transliterations or with lemma and morpheme sequences.
• I2. Union of candidate lists before rescoring. Only words in either model's top-N list are scored with the combined model. This avoids a full dictionary search while still letting each model recover candidates the other missed. The pattern suits any ensemble of retrieval-style components.
• I3. Using several predicted pronunciations instead of one. Considering multiple letter-to-phone hypotheses for a misspelling worked better than using the single best one, because the converter's N-best accuracy is much higher. Propagating N-best outputs of a noisy upstream component is generally safer than committing early.
• I4. Vowel-only sequence LM for rescoring. A 4-gram model over just the vowel phones catches errors where predicted pronunciations are most often wrong. Focusing an auxiliary LM on the most error-prone subset of symbols is a cheap, transferable idea.
• I5. N-best accuracy as an evaluation lens. Reporting accuracy at 1–4 best shows that gains grow when a user picks from a short list. This is more informative than top-1 alone for interactive correction tools.