📄 Detecting gross alignment errors in the Spoken British National Corpus

Summary

Forced alignment of transcripts to audio works well on clean speech but can fail badly on long, noisy, spontaneous recordings. The Spoken BNC, digitised from 1991–94 cassette tapes, is a hard case: tracks run 45–60 minutes, speakers overlap, transcripts treat non-speech sounds inconsistently, and dictionary pronunciations often do not match real speech. The authors aligned 46 full-tape recordings with P2FA, an HTK monophone aligner, using an extended CMUdict with British pronunciations. They then built five detectors that flag suspicious stretches of an alignment. The detectors use aligner log-probabilities, signal amplitude and segment durations. To check them, one author rated each file from 0 to 10 by spot-checking short excerpts. The main finding is that simple duration statistics, measured over words or time windows, track human judgements best. Probability-based signals help much less, because good and bad alignments produce similar Log(P) scores. This matters because large speech corpora need a way to direct human checking to the places where the aligner probably broke.

Key points

• Data: 46 one-side-of-tape recordings from the Spoken BNC, aligned with the Penn Phonetics Lab Forced Aligner (P2FA). P2FA uses monophone GMM-HMMs on 39 PLP coefficients and the CMUdict phone set, plus an extra British vowel OH.
• Human reference: a rater checked a 5-second region roughly every 60 seconds for word-level correctness. Each file then got an overall score from 0 to 10.
• The five detectors produce these features:
  • "unexpected": per-phone Log(P) compared with a prediction learned from the rest of the corpus, smoothed over 1 s.
  • "improbable": duration-normalised Log(P) for words of 4+ phones.
  • "loud" and "quiet": regions above the 97th or below the 3rd amplitude percentile, about ¼ s long, in silence or speech labels respectively.
  • "short" and "long": mean phone duration within a word falling outside 1/32 s to 1/8 s.
  • "badlength": deviation from a linear-regression phone duration model.
• Some detectors flag the same places far more often than chance, using a 5 s window. The pairs badlength/unexpected, badlength/long and long/unexpected coincided 3.6–5.1× more than expected (P < 0.001). The loud detector is anticorrelated with the speech-only detectors by design.
• Duration-based detectors (short, badlength) performed best. "short" catches runs of phones squeezed to the 3-state minimum of about 30 ms. For comparison, earlier TTS-oriented work flagged 24% of segments to catch 43% of errors.

Interesting ideas

• I1. Instead of a fixed confidence cutoff, the expected score of each unit is predicted from the rest of the corpus, leaving out the file under test, and only negative deviations from that expectation are flagged.
• I2. The strongest failure signal was a structural sign of collapse: many phones pressed to the minimum duration the model allows, which beat the model's own probability scores.
• I3. Failures tend to span whole regions rather than single units, so the deviation signal is smoothed over about a second before thresholding.
• I4. The authors measured how often detector pairs fire near each other relative to chance. This shows which detectors are redundant and which ones see different kinds of error.
• I5. Thresholds were tuned to a fixed alarm budget, about 100 flags per hour on badly aligned files, so the detectors serve as a triage tool for limited human review time.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Silver parses could be flagged when parser confidence falls well below what is typical for sentences with a similar length and tag profile, rather than using one global score threshold.
• I2 (tree): Add cheap checks for degenerate output, such as long flat chains, many tokens attached to root, or repeated default labels. These may catch gross parser failures that the parser's internal scores miss.
• I3 (tokens): Smoothing tagger or lemma anomaly scores over a window of tokens could separate one-off odd words from real breakdowns, such as code-switching or garbled OCR spans.
• I4 (learn): Apply the same chance-corrected co-occurrence test to the disagreement flags of witness parsers. That would show which witnesses add independent evidence and which merely repeat each other.
• I5 (learn): Set repair-rule and anomaly thresholds by how many items the LLM teacher tier or a human can review per batch, rather than by abstract confidence levels.