Reasoning about Pragmatics with Neural Listeners and Speakers

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D16-1125.pdf

Summary

The paper addresses contrastive scene description in a two-player reference game. A speaker must produce an utterance that lets a listener pick a target scene over a distractor. Earlier "direct" approaches learn pragmatic behavior but need training data annotated specifically for pragmatics. Earlier "derived" (rational speech acts) approaches reason about a listener but depend on hand-built grammars and small domains. Andreas and Klein combine the two. They learn a literal speaker S0 and a literal listener L0 from ordinary, non-contrastive captions of the abstract scenes dataset (Zitnick et al., 2014). Both are built from small neural modules: referent and description encoders, a choice ranker, and a feedforward conditional language model. A reasoning speaker S1 then samples candidate captions from S0 and rescores them with a weighted mix of listener and speaker probabilities. In human-played games on the test set, S1 succeeds 81% of the time, against 69% for a reimplementation of Mao et al. (2015). This shows that learned pragmatics can scale to realistic data without pragmatic supervision or domain engineering.

Key points

• Test accuracy on all pairs: S1 81%, Contrastive baseline (Mao et al., 2015) 69%, Literal S0 64%. On hard pairs the scores are 68%, 58% and 53%. All differences are significant (p < 0.05, Binomial).
• The interpolation weight λ trades accuracy (pure L0) against fluency (pure S0). With λ = 0.02, fluency stays at 4.8, the same as the literal speaker, with little loss of accuracy.
• Accuracy keeps improving as the number of sampled candidates grows up to 100, so the final system uses 100 samples.
• A "compiled" direct speaker was trained on S1's own outputs. It reaches a mean of 69% versus 66% for S0, a gain that is not significant, and stays well below S1's 83%. This suggests explicit reasoning is hard to distill into a shallow network.
• Models, human annotations and code are released at github.com/jacobandreas/pragma.