Elamur Research⌕
☰
home / papers / 2026 / Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling

Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling

Paper · 2026 · Zhenghua Wang, Yiran Ding, Changze Lv et al.

Summary

Long-context LLMs that use RoPE tend to concentrate attention on the start and end of the input. As a result, they underuse content in the middle, a problem known as "lost-in-the-middle." Earlier fixes such as Attention Buckets, MoICE and Ms-PoE mix several RoPE bases or scales. However, they need repeated forward passes or attention computations and rely on hand-picked settings.

The authors propose LPES, which gives each transformer layer its own positional-index scaling factor. Information from multiple scaled RoPEs is therefore combined within a single pass, with no fine-tuning. The per-layer factors are not searched directly, which would be intractable. Instead, they are read off a Bézier curve defined by a few control points, and a genetic algorithm optimizes those control points on a small labeled set. Experiments indicate that smooth variation of scales across depth is itself a useful inductive bias.

LPES flattens the attention distribution and improves several long-context benchmarks while keeping general abilities intact. This makes it a cheap, deployment-friendly way to reduce position bias.

Key points

  • The largest reported improvement is an 11.2% accuracy gain on a key-value retrieval dataset. LPES also runs 2.42× faster than MoICE and 1.45× faster than Ms-PoE.
  • The scale search takes about 3–4 hours on four H100 GPUs using roughly 200 examples.
  • Ablation on Vicuna-v1.5-7B with MDQA compared different numbers of control points:
    • Four control points raised accuracy from 59.9 to 63.6, with lower variance across positions, and converged in 9 epochs.
    • Brute-force search reached only 60.2 within 20 iterations.
  • Bézier parameterization beat both linear interpolation and step functions built from the same control points. Configurations with lower curvature across layers performed better.
  • Limitation: the method is evaluated only in a training-free setting. Integrating it into training or fine-tuning is left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Zhenghua Wang, Yiran Ding, Changze Lv, Yixin Wu, Tianlong Li, Zhibo Xu, Muling Wu, Tianyuan Shi, Shizheng Li, Qi Qian, Xuanjing Huang and Xiaoqing Zheng. 2026. Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1059

@inproceedings{wang2026mitigating,
  title = {{Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling}},
  author = {Zhenghua Wang, Yiran Ding, Changze Lv, Yixin Wu, Tianlong Li, Zhibo Xu, Muling Wu, Tianyuan Shi, Shizheng Li, Qi Qian, Xuanjing Huang and Xiaoqing Zheng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1059},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1059}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. 2008. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249. find
  2. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088.
  3. Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2024. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060.
  4. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023a. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595. find
  5. Hongzhan Lin, Ang Lv, Yang Song, Hengshu Zhu, Rui Yan, and 1 others. 2024. Mixture of in-context experts enhance llms’ long context awareness. Advances in Neural Information Processing Systems, 37:79573–79596. find
  6. Yuhan Chen, Ang Lv, Ting-En Lin, Changyu Chen, Yuchuan Wu, Fei Huang, Yongbin Li, and Rui Yan. 2023b. Fortify the shortest stave in attention: Enhancing context awareness of large language models for effective tool use. arXiv preprint arXiv:2312.04455. find
  7. Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, and 1 others. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zha…
  8. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024b. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  9. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024c. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  10. Dakota Mahan, Ryan Carlow, Louis Castricato, Nathan Cooper, and Christian Laforte. 2023. Stable beluga models. find
  11. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, and 1 others. 2023. Codegeex: A pre-trained model for code generation with multilingual evaluations on humanevalx. corr abs/2303.17568 (2023). arXiv preprint arXiv:2303.17568, 10.
  12. M.E. Mortenson. 1999. Mathematics for Computer Graphics Applications. G - Reference,Information and Interdisciplinary Subjects Series. Industrial Press. find
  13. Taweechai Nuntawisuttiwong and Natasha Dejdumrong. 2021. An approximation of bézier curves by a sequence of circular arcs. Information Technology and Control, 50(2):213–223. find
  14. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. Zeroscrolls: A zero-shot benchmark for long text understanding. arXiv preprint arXiv:2305.14196.
  15. Ning Shang, Li Lyna Zhang, Siyuan Wang, Gaokai Zhang, Gilsinia Lopez, Fan Yang, Weizhu Chen, and Mao Yang. 2025. Longrope2: Near-lossless llm context window scaling. arXiv preprint arXiv:2502.20082. — [arXiv:2502.20082](https://arxiv.org/abs/2502.20082) find
  16. Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding. Cornell University - arXiv,Cornell University - arXiv.
  17. Qi Sun, Edoardo Cetin, and Yujin Tang. 2025. Transformer 2 : Self-adaptive llms. arXiv preprint arXiv:2501.06252. — [arXiv:2501.06252](https://arxiv.org/abs/2501.06252) find
  18. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  19. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  20. Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed H Awadallah, Dragomir Radev, and Rui Zhang. 2021. Summˆ n: A multi-stage summarization framework for long input dialogues and documents. arXiv preprint arXiv:2110.10150. — [arXiv:2110.10150](https://arxiv.org/abs/2110.10150) find
  21. Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. 2024. Found in the middle: How language models use long contexts better via plug-and-play positional encoding. arXiv preprint arXiv:2403.04797. — [arXiv:2403.04797](https://arxiv.org/abs/2403.04797) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-mitigating-position-bias-transformers-layer/