📄 When data permutations are pathological: the case of neural natural language inference

Summary

The paper asks how much the order of training examples affects neural Natural Language Inference (NLI) models. The authors find the effect is large enough to undermine leaderboard comparisons. They build two simple attention-based architectures that use only fixed GloVe embeddings. FF/SUM is close to Parikh et al. (2016), and Bi/LSTM is a slimmed-down variant of Chen et al. (2017). Both are trained on SNLI and MULTI-NLI under many orderings of the training set: the distributed order, its reversal, one-time and per-epoch shuffling, sorting by premise, hypothesis or combined length, and sorting by model confidence, each also reversed. Only the original distributed order gives near state-of-the-art accuracy. Every other ordering costs roughly 3–4 points on SNLI and 1–6 points on MULTI-NLI, including both the standard advice to shuffle and the deep-learning habit of length bucketing. The advantage of the more complex Bi/LSTM over FF/SUM also usually shrinks under the alternative orderings. This casts doubt on whether reported architectural gains, and the benchmarks themselves, are reliable.

Key points

• Data and test-set scores. SNLI (over 570K pairs) and MULTI-NLI (over 433K pairs) were used. On test, Bi/LSTM reaches 87.12% on SNLI and 75.58% (matched) / 74.33% (mismatched) on MULTI-NLI. The authors present this as state-of-the-art for MULTI-NLI among "simple" systems, meaning embeddings only and no ensembles.
• Setup. 300-d GloVe Common Crawl 840B vectors are kept frozen, with zero vectors for OOV words. Training uses RMSprop with learning rate 0.001, batch size 512 and dropout 0.2, plus early stopping after 4 epochs without dev improvement.
• Effect of reversing the order. With the original order, dev accuracy is 85.64 (FF/SUM) and 87.53 (Bi/LSTM) on SNLI, and 72.29 and 76.43 on MULTI-NLI. Simply reversing that order drops these to 82.48 / 83.92 and 66.71 / 69.71.
• Shuffling and the architecture gap. Shuffling is the best alternative to the original order but still clearly worse. The gap between the two architectures falls from 1.89 to mostly about 0.6–1.3 on SNLI, though it reaches 2.55 under one ordering. On MULTI-NLI it falls from 4.14 to about 2.2–3.5.
• Reproducing more complex systems. Rerunning Chen et al.'s released code gave 87.77% rather than the reported 88% and took about 7 hours longer than the authors' own system. Gong et al.'s system did not finish. Code for the paper is released on GitHub (nli-data-permutations).

Interesting ideas

• I1. Treat the order of training examples as an explicit experimental factor: compare the original, reversed, shuffled-once (several seeds with σ) and per-epoch-shuffled versions, and evaluate these on dev so the test set stays untouched.
• I2. Judge an architecture by whether its advantage survives across data orderings. A gap that appears only under one particular order is likely an artefact, not a better design.
• I3. Check whether a seemingly harmless sort key, such as sentence length, is secretly correlated with the label. Sorting by that key can quietly arrange the data by class, so the authors examined class ordering separately.
• I4. Sort the data by a trained model's per-example confidence, in both directions, as a probe of how training dynamics depend on example difficulty.
• I5. Report the time and accuracy actually obtained when rerunning other people's released code, instead of copying their published numbers.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Run each training of the perceptron or gparse parser on silver data under several orderings and report the spread, so that order effects are not mistaken for real improvements.
• I2 (tree): Accept a claim that gparse beats the perceptron parser on LAS only if the margin holds under shuffled and reversed treebank orders.
• I3 (learn): Before length-bucketing or curriculum-ordering silver data, test whether the sort key correlates with genre, source treebank or witness-parser agreement, and use it as a negative control.
• I4 (answer): Order school-test or word-problem training items by solver confidence, easy-first and hard-first, to see whether the readers' accuracy depends on presentation order rather than on understanding.
• I5 (learn): When comparing against external parsers or teachers, log our own reproduced scores and runtimes so that baselines in evaluations are honest.