📄 How does Punctuation Affect Neural Models in Natural Language Inference

Summary

This paper asks whether neural Natural Language Inference (NLI) classifiers handle punctuation correctly. Such a model should ignore punctuation that carries no meaning, such as a missing final stop, and should notice punctuation that changes meaning, such as a comma that separates two list members. The authors built two altered versions of MNLI. One adds a full stop to every sentence. The other strips every non-alphanumeric character. They trained a BiLSTM, HBMP and BERT on each version and tested them on the other. The RNN models lost roughly 30 accuracy points when training and test punctuation did not match. BERT stayed almost unchanged. The authors also wrote a small set of premise/hypothesis pairs whose labels depend on commas or quotation marks. On these, every model scored near chance. Punctuation is rarely meaningful in MNLI, so models learn to treat it as noise at best. The authors argue that NLI datasets need examples where punctuation matters and that models may benefit from syntactic structure.

Key points

• The training corpus is MNLI: 433k sentence pairs from ten genres, with only five genres in training and matched/mismatched dev/test splits. Punctuation makes up about 11.5% of its tokens.
• There are three data variants: orig (unchanged), p (a final stop added everywhere) and ¬p (all non-alphanumeric characters removed).
• The models are a BiLSTM with max pooling, HBMP (Talman et al., 2019) with GloVe 840B vectors, and BERT base fine-tuned with Adam (lr 0.00002, batch 24). The two sentence encoders combine sentences as [p; h; p−h; p∗h].
• When training and test punctuation matched, the RNN models scored about .72–.73. When they did not match, accuracy fell to about .41–.44. BERT stayed between .816 and .839 in all settings.
• On the 18 hand-built punctuation-sensitive problems, accuracy was 61.1% for HBMP p, 48.8% for HBMP ¬p, 44.4% for BERT p and 48.8% for BERT ¬p. BERT's p and ¬p models gave identical predictions on the quotation cases.

Interesting ideas

• I1. The paper builds minimal pairs where only comma placement differs and the gold label flips, for example "my mother, Anna, Smith and John" against "my mother Anna, Smith and John" when counting people thanked. This exposes systems that treat punctuation as noise.
• I2. The authors separate perturbations the system should ignore, like a dropped final stop, from those it should react to, like a meaning-bearing comma. They then test invariance and sensitivity as two distinct hypotheses.
• I3. They train on one punctuation variant and test on another, which shows a model's dependence on surface tokens cheaply and without new annotation.
• I4. One contrast hinges on "No, god is good" versus "no god", where a comma decides between a vocative/interjection and a negative quantifier. This tiny change has a large logical effect.
• I5. The paper probes quotation scope by contrasting "I hear John says 'come here'" with "I hear 'John says come here'", where quote marks change who is speaking. Both HBMP models failed on the second form, and BERT gave the same answers whether trained with or without punctuation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): a similar contrast suite could check whether the parser chooses apposition or conjunction correctly depending on commas, before frames are built on top.
• I2 (learn): evaluation could include both kinds as controls, so that answers must stay stable under the first kind and must change under the second.
• I3 (learn): stripping or normalising punctuation in held-out inputs is an inexpensive negative control for spotting pipeline stages that lean on punctuation by accident.
• I4 (frames): such pairs could test whether frame building and speech-act detection keep an interjection separate from a quantifier that negates the subject.
• I5 (frames): similar pairs could check that reported-speech frames attach the speaker and the quoted content according to where the quotation marks fall.