📄 The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations

Summary

The paper reports on a shared competition at RepEval 2017. Its goal was to measure how well neural sentence encoders capture meaning. Each sentence had to be compressed into a single fixed-length vector, and the vectors were then used for three-way natural language inference (entailment, neutral, contradiction) on the MultiNLI corpus. Mechanisms that let the two sentences in a pair look at each other, such as cross-sentence attention or memory over token sequences, were banned. This kept the comparison focused on the encoders. Two test conditions were run. In the matched condition, test genres also appear in training. In the mismatched condition, the five test genres were never seen in training. Five teams took part, and every one of them outperformed the BiLSTM and CBOW baselines. All the submitted systems were built on BiLSTMs. The strongest single model reached 74.5% matched accuracy using stacked BiLSTMs with residual (shortcut) connections. The gap between matched and mismatched accuracy turned out to be small. Results were also fairly even across the linguistic-phenomenon subsets. The organizers read this as evidence that the encoders learned fairly genre-independent representations, while NLI remains well below human level.

Key points

• MultiNLI has about 393k sentence pairs (392,702 for training) drawn from ten genres of written and spoken English. Only five genres (FICTION, GOVERNMENT, SLATE, TELEPHONE, TRAVEL) come with training data. Each genre has 2,000 dev and 2,000 test pairs.

• Scoring ran on Kaggle in Class. The public leaderboard showed only a random 25% of test labels. Final scores came from the hidden 75%, using each team's two chosen systems.

• Final matched / mismatched accuracy:

  System	Matched	Mismatched
  Chen et al. ensemble ("alpha")	74.9%	74.9%
  Nie and Bansal, single model	74.5%	73.5%
  BiLSTM baseline	67.0%	67.6%

• Competition rules:

  • Training on the dev sets was forbidden.
  • Any publicly available outside data was allowed, and SNLI was explicitly encouraged.
  • Teams could submit at most two runs per day during a two-week evaluation window.

• Teams also supplied vectors for automatically generated probe sentences. In these, vector similarity separated negation, subject identity and tense, but continuous tenses were not reliably distinguished. Behaviour on the probes did not clearly track overall accuracy.

Interesting ideas

• I1. Holding out whole genres from training (the mismatched split) shows whether a model learned general meaning or just the style of its training text.
• I2. Banning cross-sentence interaction keeps the evaluation on the component of interest, so alignment tricks cannot mask weak representations.
• I3. Minimal-pair probe sentences that vary only negation, role assignment or tense show which distinctions a representation actually encodes.
• I4. Showing only a small public slice of the test labels and capping daily submissions discourages tuning to the evaluation set.
• I5. Reporting annotator agreement per genre next to model accuracy gives a realistic ceiling and flags genres where the gold labels themselves are shaky.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluate each trained component on a genre deliberately absent from its silver data, and report the matched-vs-unseen gap next to the main score.
• I2 (answer): Build the world of each text independently, then decide entailment-style questions by comparing the two worlds, so errors point at world construction rather than the matcher.
• I3 (frames): Generate such minimal pairs automatically and check that the resulting frames differ exactly on polarity, agent/patient or tense, catching silent role swaps.
• I4 (learn): Keep a hidden portion of the gold test data that repair rules and self-play loops never see, consulted only for occasional final checks.
• I5 (learn): Store agreement among witness parsers per genre and use it to set expectations and to down-weight low-consensus silver data.