📄 Understanding and Bridging the Modality Gap for Speech Translation

Summary

End-to-end speech translation (ST) has little parallel training data, so systems often borrow knowledge from text machine translation (MT) through multi-task learning with a shared encoder-decoder. Fang and Feng ask why ST still trails MT. They quantify a "modality gap" as the cosine distance between final decoder states computed from speech and from its transcript. On MuST-C En→De, the gap is mostly small under teacher forcing, apart from a long tail of hard tokens. At inference, however, it widens with every decoding step, because ST and MT condition on diverging self-generated prefixes. The authors frame this compounding effect as a form of exposure bias.

Their method, CRESS, applies scheduled sampling to both tasks, so training contexts mix gold and model-predicted tokens. It also adds a KL-divergence term that pulls the ST and MT output distributions together. A token-level adaptive weighting further emphasizes tokens with large gaps. CRESS outperforms a strong multi-task baseline in all eight MuST-C directions, with the largest benefit on long sentences. The work offers both a diagnostic tool and a practical training recipe for speech-to-text transfer under data scarcity.

Key points

• Model: a pre-trained HuBERT encoder plus two convolutional layers that shorten the speech sequence. These feed a Transformer translation model, pre-trained on MT data, whose layers are fully shared between speech and text inputs.
• The modality gap is defined as 1 − cosine similarity of last-decoder-layer representations. On the MuST-C En→De dev set it is below 10% in most cases.
• Under greedy search the gap grows steadily with decoding steps. Beam search (beam size 8) slows this growth, and teacher forcing keeps the gap flat across steps.
• Average improvements over the multi-task learning baseline across eight MuST-C directions: +1.8 BLEU (base setting) and +1.3 BLEU (expanded setting).
• Code is released at github.com/ictnlp/CRESS (ACL 2023, Long Papers).