📄 Attention Strategies for Multi-Source Sequence-to-Sequence Learning

Summary

The paper looks at how a single recurrent decoder should combine attention when it reads several encoders at once, possibly of different modalities. The usual practice concatenates per-encoder context vectors. That leaves the weighting of the sources to later layers and gives no explicit view of how much each input is used. The authors propose two alternatives. Flat combination computes one softmax over the hidden states of all encoders after projecting them into a shared space. Hierarchical combination first builds a context vector for each encoder, then applies a second attention over those vectors. Both can optionally include a sentinel adapted to GRUs, which lets the decoder rely on its own state. The strategies were tested on WMT16 multimodal translation (Multi30k) and WMT16 automatic post-editing, implemented in Neural Monkey. Hierarchical attention without the sentinel gave the best multimodal scores and the fastest convergence. In APE, all variants slightly beat the do-nothing baseline but did not differ significantly from each other. The work matters because it makes the contribution of each source interpretable and works with conditional GRU decoders, where concatenation breaks down.

Key points

• Data:
  • Multi30k has 29,000 training, 1,014 validation and 1,000 test instances (image, English caption, German translation).
  • The WMT16 APE data has 12,000 / 2,000 / 1,000 English–MT German–post-edited German triplets from the IT domain.
• Multimodal translation results:
  • Hierarchical attention (no sentinel, unshared projections) reached BLEU 32.1 and METEOR 49.1, against 31.4 / 48.0 for concatenation.
  • The text-only baseline (BLEU 33.0) was still not surpassed.
• Conditional GRU effect: on average, conditional GRU units added about 1.5 BLEU to the new strategies. With concatenation, however, they cut about 5 BLEU.
• APE results:
  • All models reached HTER of roughly 24.1–24.4, versus 24.8 for leaving the MT output unchanged.
  • The differences between combination strategies were not significant.
• Setup:
  • Bidirectional GRU encoders with 300 units per direction and 300-dimensional embeddings.
  • A 500-unit attention space and Adam with learning rate 10⁻⁴.
  • A frozen VGG 16 for images and a 20,000-unit shared BPE vocabulary.

Interesting ideas

• I1. Explicit source weights: making the decoder produce weights over sources, rather than mixing them implicitly, yields per-step weights you can inspect. In the paper's visualization, the image encoder lights up mainly on content words. This is handy for diagnosing whether an auxiliary input, such as metadata or a second language, is actually used.
• I2. Two-level attention: attending first inside each source and then across sources is a cheap, modular way to fuse heterogeneous inputs of different dimensionality. Each source is only projected into a common space at the top level.
• I3. Sentinel option: a sentinel entry lets the model "attend to nothing" and lean on its own language-model-like state. This is useful when some inputs are often uninformative for the current output step.
• I4. Edit-operation targets for post-editing: the APE targets are encoded as edit operations on the MT output rather than as full sentences. This stops the model from freely paraphrasing, which suits any correction task where the input is mostly right.
• I5. Sharing projections is not free: the paper tested whether energy and context computations can share one projection. Sharing brought no gain and made training dramatically slower, a reminder to test such parameter-saving shortcuts empirically.