📄 A Hybrid Morphologically Decomposed Factored Language Models for Arabic LVCSR

Summary

Arabic attaches many prefixes and suffixes to stems, so the number of distinct word forms grows very large. This weakens n-gram estimates and hurts large vocabulary continuous speech recognition (LVCSR). The authors combine two known remedies: splitting words into morphemes, and factored language models (FLMs), which back off to other word features when a word sequence was never seen.

Training text was annotated with MADA 2.0, and each unit received four factors:

• surface form (W),
• lexeme (L),
• morphological description (M),
• pattern (P), derived from a Sebawai root.

The best factor combination and backoff path were found with the genetic GA-FLM tool and then hand-tuned. The resulting FLMs rescore N-best lists in the last pass of a three-pass GALE recognizer.

The hybrid system, an FLM over partially decomposed units, gave relative WER reductions of 3.5–7.0% over plain full-word systems and 1.0–2.0% over decomposed systems without factors. Gains held at both 70k and 256k lexicons. The work shows that the two remedies complement each other for morphologically rich languages.

Key points

• Setup:
  • FLMs were built with the SRILM-FLM extensions, limited to two previous time slots (trigram-like) because of memory limits.
  • Training data: 206M running words.
  • Acoustic models: 1100h of audio, ML training followed by MPE.
• Choosing the topology:
  • On a 40-minute GALE dev set, the chosen FLM conditioned on W, M and L of the two previous units and reached 19.9% WER, against 20.4% for the trigram-equivalent baseline.
  • This was not the lowest-perplexity variant.
  • Rescoring worked best with 10-best lists.
• 70k full-word system (dev07):
  • Full-word FLM rescoring lowered WER from 16.5% to 15.7%, a 4.8% relative reduction against tri-gram lattice rescoring.
  • Decomposing the full-word N-best lists did not help.
• 70k partially decomposed system (20k words + 50k morphemes):
  • Best result was 14.3% with an FLM over partially decomposed units, against 14.7% for the new baseline.
  • This is 2.2% absolute below the original full-word baseline.
• 256k partially decomposed system:
  • With FLM rescoring: dev07 13.9%, eval07 15.9%.
  • 4-gram baseline: 14.2% / 16.1%. Traditional 256k full-word system: 14.9% / 16.5%.
  • OOV rates fell sharply with decomposition, e.g. on eval07 at 70k from 4.82% (full words) to 1.94% (partial).

Interesting ideas

• I1. Treat each token as a bundle of features (form, lexeme, morph tag, pattern), so an unseen word sequence can still be scored through its more general features.
• I2. Keep frequent words whole and split only the rarer ones into morphemes, which cuts OOV rates without making sequences too fragmented.
• I3. Rescoring gave no gain when the candidate lists were produced in full-word units and only decomposed afterwards, so the candidate generator and the rescorer should use the same units from the start.
• I4. The model with the lowest perplexity was not the one with the lowest WER, so model choice was made on the downstream task.
• I5. A genetic algorithm searched the large space of factor combinations and backoff paths, followed by manual fine-tuning.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): the tagger and lemmatizer could back off from rare surface forms to lemma and morph-class contexts, with each fallback step logged so the decision stays explainable.
• I2 (tokens): the tokenizer could split rare compounds, hyphenations and contractions while leaving common words intact, reducing unknown tokens for the parser.
• I3 (tree): parse candidates for gparse or the perceptron reranker should come from a tokenization that matches the scoring model, not one converted after the fact.
• I4 (learn): parser or tagger variants should be chosen by end-task accuracy (frames, answers), not only by intrinsic scores on held-out text.
• I5 (learn): a similar automated search could explore feature templates or backoff orders for the perceptron models, with a human pass on the shortlisted configurations.