📄 Language as a Latent Variable: Discrete Generative Models for Sentence Compression

Summary

The paper reframes sentence compression as a generative problem. Instead of training only a discriminative sequence-to-sequence mapping, it treats the short summary as a hidden, discrete string that the model must infer. In the auto-encoding sentence compression (ASC) model, a pre-trained language model acts as a prior that proposes a latent compression, and a decoder then reconstructs the full sentence from it. Inference uses a variational auto-encoder. Because the latent variable is discrete text, the reparameterisation trick cannot be used, so the authors train with REINFORCE plus variance-reduction baselines. A pointer network serves as the inference network, which keeps samples extractive.

To stabilise training, they add a supervised forced-attention sentence compression (FSC) model. FSC shares the pointer network with ASC but can also generate words from the full vocabulary. Trained jointly on labelled and unlabelled data from the Gigaword compression task, the combination reaches state-of-the-art results with full supervision. With only a fraction of the labelled data, it stays competitive with earlier supervised systems. This shows that unlabelled text can be used effectively for sequence-to-sequence tasks.

Key points

• ASC is built from four RNNs: a bidirectional LSTM encoder, an LSTM compressor (pointer network), a soft-attention decoder, and a pre-trained language model used as the prior p(c).
• The training objective is a modified variational lower bound. It combines reconstruction log-likelihood with a KL term scaled by λ = 0.1 in all ASC experiments, because the language-model prior otherwise pushes compressions to be too short.
• Gradients for the compression network are estimated with REINFORCE using two baselines: a centred learning signal b and an input-dependent baseline b(s) implemented as an MLP.
• FSC combines copying from the source (pointer) with a softmax over the whole vocabulary. This lets it produce abstractive compressions while sharing the pointer network that links supervised and unsupervised training.
• On the Gigaword compression benchmark, the joint ASC+FSC model outperforms previously published results when trained on all supervised data. In semi-supervised settings, it matches earlier benchmarks while using much less labelled data.

Interesting ideas

• I1. Text as the bottleneck: The auto-encoder's hidden code is a readable sentence rather than a dense vector, so the compression itself becomes the learned representation. This could suit any task where an interpretable intermediate string is wanted, such as simplification, keyword extraction, or query rewriting.
• I2. Restricting the sample space with a pointer network: Limiting latent samples to words from the source shrinks the action space from the whole vocabulary to the sentence length. This makes sampling-based training feasible early on and is a useful trick for any discrete-latent or RL-style text model with huge vocabularies.
• I3. A supervised "teacher" sharing parameters with an unsupervised model: A small labelled model gives the unsupervised learner sensible starting behaviour instead of a random policy, while unlabelled data improves the shared component in return. This is a practical recipe for low-resource settings such as Ukrainian NLP, where parallel pairs are scarce but raw text is plentiful.
• I4. The KL weight as a length control: The authors observe that λ influences how aggressively sentences are compressed. A single scalar could therefore serve as a tunable dial for output length or regularisation strength in other generative text models.
• I5. Reconstruction as a reward: Rewarding a compression by how well the original can be rebuilt from it replaces task-specific metrics like ROUGE with a label-free signal. The same idea could train summarisers or paraphrasers on unannotated corpora.