Elamur Research⌕
☰
home / papers / 2025 / Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding

Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding

Paper · 2025 · Zikai Xiao, Ziyang Wang, Wen Ma et al.

Summary

Long-context LLMs often perform worse as inputs grow, even when the input fits inside their nominal context window. The authors analyze this from the decoding side and describe Posterior Salience Attenuation (PSA). As context length increases, the gold token's rank-based salience score drops. Yet the correct answer usually stays near the top of the vocabulary ranking. Error analysis suggests models over-favor "proximal" tokens near the query.

To address this, they propose Positional Contrastive Decoding (PCD), a training-free method. It builds a local-aware attention variant by over-rotating RoPE's lower-frequency components, using a reduced base and a smooth transition across frequency blocks. Its logits are then subtracted from the standard long-aware logits, restricted to the top-ranked candidate tokens. A single-layer simulation indicates this slows long-range attention decay.

Experiments use Llama3-8B variants on RULER, InfiniteBench and LongBench. PCD mostly beats decoding baselines and calibration baselines, though a few settings show small drops. This makes it a cheap inference-time option for better use of long contexts.

Key points

  • Motivating observation: in a key-value retrieval task, the gold token typically ranked within the top 8 candidates despite a vocabulary of about 128k. This motivates boosting already-high-ranked answers rather than retraining.
  • Models, benchmarks and baselines:
    • Models: Llama3-8B-8k plus Gradient AI's long-context Llama3-8B-262k and Llama3-8B-1048k.
    • Benchmarks: RULER, InfiniteBench and LongBench (LongBench tested on the 262k model only).
    • Baselines: greedy, beam search, DoLa, MsPoE, Segment-Reranking and Rephrasing.
  • Main gains on the 262k model:
    • InfiniteBench KV retrieval at 8k improved from 72.0% to 79.0%.
    • RULER Variable Tracking at 4k improved from 74.02 to 81.80 F1.
    • LongBench average reached 26.87, versus 25.98 for the base model and 26.27 for MsPoE.
  • Hyperparameter ablation (InfiniteBench):
    • Best contrast coefficient: β = 2.5.
    • Best base ratio: B/B′ = 10⁴.
    • Transition coefficient α (about 0.1–0.2) and top-γ (about 20–30) were stable across settings.
  • Limitations:
    • PCD does not enlarge the attention window.
    • Gains on short-text tasks are small.
    • Effectiveness may depend on the positional encoding scheme.
    • Results were not uniformly positive, e.g., a slight drop for the 8k model at 8k on KV retrieval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Science and Technology of ChinaHefei · China108 works · 112,698 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🔬Zhejiang LabHangzhou · China7 works · 1,596 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Zikai Xiao, Ziyang Wang, Wen Ma, Yan Zhang, Wei Shen, Yan Wang, Luqi Gong and Zuozhu Liu. 2025. Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-short.58

@inproceedings{xiao2025mitigating,
  title = {{Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding}},
  author = {Zikai Xiao, Ziyang Wang, Wen Ma, Yan Zhang, Wei Shen, Yan Wang, Luqi Gong and Zuozhu Liu},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-short.58},
  url = {https://doi.org/10.18653/v1/2025.acl-short.58}
}

References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. 2024. Make your llm fully utilize the context. arXiv preprint arXiv:2404.16811. find
  2. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Associat… find
  3. Jinhe Bi, Danqi Yan, Yifan Wang, Wenke Huang, Haokun Chen, Guancheng Wan, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, et al. 2025. CoT-Kinetics: A theoretical modeling assessing lrm reasoning process. arXiv preprint arXiv:2505.13408. — [arXiv:2505.13408](https://arxiv.org/abs/2505.13408) find
  4. Ta-Chung Chi, Ting-Han Fan, Peter J Ramadge, and Alexander Rudnicky. 2022. Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35:8386–8399. find
  5. Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. 2024. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations.
  6. TUNING ON SYNTHETIC DATA. From artificial needles to real haystacks: Im-proving retrieval capabilities in llms by fine-tuning on synthetic data. find
  7. Amanda Dsouza, Christopher Glaze, Changho Shin, and Frederic Sala. 2024. Evaluating language model context windows: A" working memory" test and inference-time correction. arXiv preprint arXiv:2407.03651. — [arXiv:2407.03651](https://arxiv.org/abs/2407.03651) find
  8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  9. Amey Hengle, Prasoon Bajpai, Soham Dan, and Tanmoy Chakraborty. 2024. Multilingual needle in a haystack: Investigating long-context behavior of multilingual large language models. arXiv preprint arXiv:2408.10151. find
  10. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024a. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling.
  11. Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, et al. 2024b. Found in the middle: Calibrating positional attention bias improves long context utilization. arXiv preprint arXiv:2406.16008. find
  12. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  13. Taiming Lu, Muhan Gao, Kuai Yu, Adam Byerly, and Daniel Khashabi. 2024. Insights into llm long-context failures: When transformers know but don’t tell. arXiv preprint arXiv:2406.14673. find
  14. Alexander Peysakhovich and Adam Lerer. 2023. Attention sorting combats recency bias in long context language models. arXiv preprint arXiv:2310.01427. — [arXiv:2310.01427](https://arxiv.org/abs/2310.01427) find
  15. Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations. find
  16. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.
  17. Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś. 2024. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36. find
  18. Proof. For each 2D subspace V j = R 2 , let (1) (2) (q j , q j ) = R θ j (m)(W q x m ) j and similarly for k j . Using the inner product invariance under rotation: Yijiong Yu. 2023. " paraphrasing the original text" makes high accuracy long-context qa. arXiv preprint arXiv:2312.11193. find
  19. S j (k) = ⟨R θ j (m)(W q x m ) j , R θ j (n)(W k x n ) j ⟩ = ⟨(W q x m ) j , R θ j (k) ⊤ (W k x n ) j ⟩ Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. 2024a. Infinite bench: Extending long context evaluation beyond 100k to… find
  20. Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. 2024b. Found in the middle: How language models use long contexts better via plug-and-play positional encoding. arXiv preprint arXiv:2403.04797. — [arXiv:2403.04797](https://arxiv.org/abs/2403.04797) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2025-mitigating-posterior-salience-attenuation-long/