📄 No-Worse Context-Aware Decoding: Preventing Neutral Regression in Context-Conditioned Generation

Summary

LLMs given external passages, such as retrieved evidence in RAG, sometimes change an answer they already had right, even when the passage adds nothing useful. The authors call this failure "neutral regression". They measure it as the accuracy lost on baseline-correct questions when the supplied context agrees with the answer. Existing contrastive decoders (CAD, AdaCAD, CoCoA) always reweight logits, so they cannot promise to leave such answers untouched.

The paper proposes NWCAD, a decode-time adapter that runs a with-context and a no-context forward pass in parallel. At each token, a two-stage gate makes one of three choices:

• It copies the no-context logits exactly when both streams agree and the no-context stream is confident.
• It uses the with-context logits when that stream is decisive.
• Otherwise, it applies a CAD-style fallback.

Under greedy decoding, the backoff branch provably reproduces the no-context token. Evaluation uses controlled slices that separate do-no-harm cases from genuinely helpful contexts, plus several QA benchmarks. NWCAD avoids neutral regression while keeping strong accuracy when context helps, adds almost no latency, and needs no extra models.

Key points

• Gating signals. Agreement between streams is measured by Jensen–Shannon divergence, approximated over the union of each stream's top-50 tokens. Confidence is the top-1 vs. top-2 probability margin. The thresholds τ, κ_pri and κ_ctx tune how conservative the decoder is.
• Evaluation data. Controlled slices (Restate-hard, Distractor-hard, Helpful) and benchmarks including NQ-SYNTH, NQ-SWAP, HotpotQA-distractor, ToFuEval and ExpertQA, tested on several backbones including Llama-3.1-8B.
• Main gains.
  • Stage 2 adds about 5.2% average accuracy over the Stage 1-only variant.
  • Wrapping CAD, AdaCAD or CoCoA with NWCAD improves controlled-slice accuracy by roughly 7 to over 40 points, with the largest gains for CAD and CoCoA.
• Fallback rarely needed. The fallback fires on only about 1–2% of tokens, and removing it barely changes results. Most of the benefit comes from choosing between the no-context and with-context streams.
• Cost and limitations. Latency stays within about 2% of the base decoder and is sometimes faster. Thresholds were tuned on Llama-3.1-8B only. The method requires logit access, so it does not apply to black-box APIs. Code: github.com/CastGryff/NWCAD.