📄 Aligning Sentences in Parallel Corpora

Summary

Brown, Lai and Mercer address sentence alignment: pairing each sentence in a bilingual corpus with its translation, even when one sentence becomes two or text is dropped on one side. Their data is the English-French Canadian Hansard proceedings (1973–1986), with roughly 3.5M English and 3.7M French sentences. The method has two stages. First, structural comments in the transcripts (speaker names, timestamps, session markers) serve as anchors and are matched by dynamic programming, which splits the corpora into short subsections. Second, sentences inside each subsection are aligned with a hidden Markov model. The model looks only at sentence lengths in tokens and at paragraph markers, never at the words themselves. Alignments are built from "beads" (1–1, 1–2, 2–1, 1–0, 0–1 groupings), and parameters are estimated with EM. A hand check of 1000 aligned pairs found 6 errors. Because the method ignores vocabulary, it is fast enough for very large corpora. It laid groundwork for building the sentence-pair data that statistical machine translation depends on.

Key points

• Corpus statistics: the English portion contains 3,510,744 sentences totaling 85,016,286 tokens, while the French portion contains 3,690,425 sentences totaling 97,857,452 tokens. Average sentence length is 24.2 tokens for English and 26.5 for French, so French runs about 9.5% longer.
• Anchor alignment:
  • Major anchors are aligned first, using costs of 0 for an exact match, 10 for a mismatch and 5 for a deletion. Names are scored by letter-level edit distance, scaled to the range 0–10.
  • Sections whose minor anchors (e.g., Mr. Speaker / M. le Président) do not line up are discarded, which removes about 10% of the data.
  • The remaining subsections average about ten sentences.
• EM estimates: about 91% of English sentences and 98% of paragraph markers map one-to-one to French counterparts.
• Simulated error rates:
  • About 0.9% with anchors and paragraph markers.
  • 2.0% with paragraph markers only.
  • 2.3% with anchors only.
  • 3.2% with neither.
• Full run: 10 days on an IBM 3090 with 16 MB of virtual memory, producing 2,869,041 one-to-one beads. A manual check of 1000 of them found 6 incorrect.

Interesting ideas

• I1. Length is a strong signal on its own. Sentence length, with no lexical features, is enough for a cheap and language-agnostic baseline aligner. This makes it attractive for low-resource language pairs where no bilingual dictionary exists.
• I2. Use document metadata as anchors. Speaker tags, timestamps and section headers can be aligned first to cut a huge alignment problem into small, safe chunks. This applies to any parallel or comparable collection that has markup, such as subtitles, legal codes or parliamentary records.
• I3. Throw away inconsistent chunks. Sections where secondary anchors disagree are rejected outright rather than force-aligned. Sacrificing about 10% of the data for higher precision is a sensible trade-off whenever the corpus is large.
• I4. Estimate accuracy by simulation. The authors generate synthetic corpora from the fitted model and measure how well the aligner recovers the known alignment. They also switch features off one at a time to measure each one's contribution. This lets you predict accuracy before costly manual annotation.
• I5. Use information theory as a feasibility check. Comparing the entropy of the bead process (1.26 bits) with the mutual information between sentence lengths (1.85 bits) shows whether a signal can, in principle, support alignment. This is a quick test worth reusing for other weak features.