Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models
Paper · 2026 · Hyunjong Ok and Jaeho Lee
Summary
This paper explains why decoder-only language models are sensitive to how a multiple-choice prompt is arranged. When the passage comes first, followed by the question and options (CQO), models do much better than when the passage comes last (QOC). The authors tested 21 decoder-only LLMs from the LLaMA 3, Qwen 2.5/3 and Gemma 2 families (0.5B–9B parameters, base and instruct variants) on LogiQA, RACE-H, RACE-M and SciQ. They then checked three possible explanations through controlled experiments.
Two explanations failed. Unfamiliarity with the format from training data did not account for the gap, and neither did the models forgetting options placed mid-prompt. The evidence instead points to the causal attention mask. In QOC, option tokens are encoded before the context appears, so their representations carry no information from it.
Several experiments support this account:
- Bidirectional architectures show almost no gap.
- QOC accuracy roughly equals accuracy with the context removed entirely.
- Targeted interventions on the option–context pathway move accuracy in the predicted directions.
The work turns a known prompt-sensitivity effect into a mechanistic explanation. It also suggests cheap fixes for inference time.
Key points
- Size of the gap: averaged over 21 models, accuracy falls from 69.26% (CQO) to 54.54% (QOC), a gap of about 14.7 points.
- Architecture comparison: Flan-T5 (encoder-decoder) shows a 2.30-point gap, and BERT, RoBERTa and ALBERT (encoder-only) show 0.02.
- Rejected explanations:
- Base and instruct models have nearly identical gaps.
- Five-shot ICL lifts QOC by only about 3.1 points.
- Option recall under QOC is as good as or better than under CQO.
- Attribution and modulating factors: Gradient×Input attribution gives the context 0.797 of the total attribution in CQO but only 0.335 in QOC. Longer contexts widen the gap (24.8% on RACE-M vs. 6.2% on LogiQA). Correct answers in position A suffer more (22.4%) than those in position D (9.9%).
- Interventions:
- Blocking option-to-context attention in CQO costs 26.8 points.
- Activation patching improves QOC by 6.0 points.
- Repeating the options after the context (QOCO) improves QOC by 8.2 points.
- CoT prompting shrinks the gap from 14.72 to 7.47 points.
Fields, topics and concepts
📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works
🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works
Cite
Cite as Hyunjong Ok and Jaeho Lee. 2026. Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models. Findings of the Association for Computational Linguistics: ACL 2026.
@inproceedings{ok2026lost,
title = {{Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models}},
author = {Hyunjong Ok and Jaeho Lee},
booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
year = {2026},
doi = {10.18653/v1/2026.findings-acl.1921},
url = {https://doi.org/10.18653/v1/2026.findings-acl.1921}
}References
- Samira Abnar and Willem Zuidema. 2020. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190–4197.
- Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2024. Make your llm fully utilize the context. Advances in Neural Information Processing Systems, 37:62160–62188. \#\# Conclusion
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/ok-2026-lost-prompt-order-revealing-limitations/