A Decomposable Attention Model for Natural Language Inference

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D16-1244.pdf

Summary

Natural language inference (NLI) means deciding whether a hypothesis follows from a premise, contradicts it, or is neutral. Earlier neural approaches built large sentence encoders from LSTMs or CNNs. Those models have millions of parameters and read text one word after another. The authors argue that most of the task can be solved by matching small fragments of the two sentences and then combining the evidence. Their model has three feed-forward stages that are trained together. Attend softly aligns word embeddings across the sentence pair. Compare turns each word and the material aligned to it into a vector. Aggregate sums these vectors and classifies the result. The basic version ignores word order entirely. An optional intra-sentence attention step adds a distance-based bias, which gives the model a small amount of ordering information. On SNLI, this design matched or beat the best published accuracy with about ten times fewer parameters than the strongest LSTM model. It can also run in parallel across sentence positions. The results suggest that for this task, pairwise alignment matters more than a representation of the whole sentence.

Key points

• Data: the paper uses the SNLI corpus with three classes. After dropping pairs that have no gold label, it has 549,367 training, 9,842 development and 9,824 test pairs.
• Results: the vanilla model reaches 86.3% test accuracy with 382K parameters, not counting embeddings. Adding intra-sentence attention raises this to 86.8% with 582K parameters. For comparison, the LSTMN of Cheng et al. scores 86.3% with 3.4M parameters, and the mLSTM scores 86.1% with 1.9M.
• Efficiency: each attention score is computed as F(a_i)ᵀF(b_j), so the network F runs only ℓ_a + ℓ_b times instead of once per word pair. The total cost is O(ℓd² + ℓ²d), the same as an LSTM with attention. When sentence length is below the embedding size, the cost drops to that of a plain LSTM, and unlike an LSTM it can be parallelized across sentence length.
• Setup: the model uses fixed 300D GloVe vectors projected down to 200D, and out-of-vocabulary words are hashed into 100 random vectors. The networks have 2 layers of 200 ReLU units. Training uses Adagrad with dropout 0.2, implemented in TensorFlow.
• Error analysis: most gains over earlier models come from neutral pairs, and most losses come from contradictions. The model still fails when the answer depends on word order, on counting, or on a single opposing word such as closes vs open.