Elamur Research⌕
☰
home / papers / 2026 / Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities

Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities

Paper · 2026 · Alberto Purpura, Li Wang, Sahil Badyal et al.

Summary

Current instruction-following benchmarks often use constraints with little practical value, tie constraints to the specific prompt task so that compliance and task success get mixed together, and rarely test more than a handful of instructions at once. To tackle these issues, the authors introduce MOSAIC, a benchmark built through dynamic generation, whose name reflects a modular and synthetic approach to evaluating instruction compliance. Each prompt combines one of four business writing tasks, one of eight products or services, and a list of 1 to 20 constraints sampled from a pool of 21 application-oriented rules. Because the constraints are a separate, reusable module, they can be paired with other tasks, and regenerating the data guards against leakage. Rule-checkable constraints are scored automatically. Semantic and business/legal constraints are scored by an LLM judge that was validated against human annotators. The evaluation covers models from several families, including Llama, Qwen3, Deepseek, Mixtral, Gemini and Claude. It shows that compliance depends strongly on the type, number and position of constraints. It also reveals helpful and conflicting interactions between instructions, as well as model-specific positional biases. These diagnostics help pinpoint failures in systems that require strict rule adherence.

Key points

  • The dataset has 4,000 prompts averaging 10.5 constraints each, which is more than double the density of benchmarks like ComplexBench, IFEval or InFoBench. Prompts average about 222 words, and the constraints span 5 groups and 21 types.
  • The tasks are a marketing email, a product review, an FAQ entry and an internal memo. Crossed with 8 product/service types, they give 32 generation variants.
  • Constraint position matters in different ways for different models:
    • Llama, Qwen3 and Deepseek show a primacy effect, following early constraints more reliably.
    • Mixtral-8x-7B, Gemini and Claude favor later constraints, a recency effect.
    • Claude and Gemini spike sharply at position 19 and then drop at position 20.
  • The number of constraints also shapes compliance:
    • With 1–6 constraints, Qwen3 is the most compliant model.
    • Between 7 and 15 constraints, results become erratic for all models.
    • Beyond 15, compliance falls for nearly all models, most sharply for reasoning models such as Deepseek-R1 8b and Qwen3.
  • The code is public (CapitalOne-Research/llm-instruction-following-compliance on GitHub). Prompts are English only, but the authors suggest automatic translation could extend the benchmark to other languages, such as Ukrainian.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Capital OneTysons Corner · United States6 works · 16 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Alberto Purpura, Li Wang, Sahil Badyal, Gene Beaufrand and Adam Faulkner. 2026. Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.62

@inproceedings{purpura2026deconstructing,
  title = {{Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities}},
  author = {Alberto Purpura, Li Wang, Sahil Badyal, Gene Beaufrand and Adam Faulkner},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.62},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.62}
}

References

19 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. Introducing the model context protocol. find
  2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  3. Rudolph Flesch. 1948. A new readability yardstick. Journal of applied psychology, 32(3):221. find
  4. Cristina Garbacea and Qiaozhu Mei. 2022. Why is constrained neural language generation particularly challenging? arXiv preprint arXiv:2206.05395. — [arXiv:2206.05395](https://arxiv.org/abs/2206.05395) find
  5. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  6. Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Jiaqing Liang, and Yanghua Xiao. 2024a. Can large language models understand real-world complex instructions? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18188–18196. find
  7. Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, Shruti Bhosale, Chenguang Zhu, Karthik Abinav Sankararaman, Eryk Helenowski, Melanie Kambadur, Aditya Tayade, Hao Ma, Han Fang, and Sinong Wang. 2024b. Multi-if: Benchmarking llms on mul… — [arXiv:2410.15553](https://arxiv.org/abs/2410.15553) find
  8. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024a. Mixtral of experts. arXiv preprint arXiv:2401.04088. find
  9. Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024b. Followbench: A multi-level fine-grained constraints following benchmark for large language models.
  10. Bohdan Kovalevskyi. 2024. Ifeval-extended: Enhancing instruction-following evaluation in large language models through dynamic prompt generation. Journal of Artificial Intelligence General science (JAIGS) ISSN: 3006-4023, 5(1):513–524. find
  11. Jinnan Li, Jinzhe Li, Yue Wang, Yi Chang, and Yuan Wu. 2025. StructFlowBench: A structured flow benchmark for multi-turn instruction following. In Findings of the Association for Computational Linguistics: ACL 2025, pages 9322–9341, Vienna, Austria. Association for Computational Linguistics. find
  12. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  13. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172.
  14. Yile Liu, Ziwei Ma, Xiu Jiang, Jinglu Hu, ChangJing ChangJing, and Liang Li. 2025. MaXIFE: Multilingual and cross-lingual instruction following evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14252–14332, Vienna, A… find
  15. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. Infobench: Evaluating instruction following ability in large language models.
  16. Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, and Minlie Huang. 2024. Benchmarking complex instruction-following with multiple constraints composition. find
  17. Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Lu Xiangju, Junmin Zhu, and Wei Zhang. 2025. LIFBench: Evaluating the instruction following performance and stability of large language models in long-context scenarios. In Proceedings of the 63rd Annual Meeting of the Association for Compu…
  18. Zhihan Zhang, Shiyang Li, Zixuan Zhang, Xin Liu, Haoming Jiang, Xianfeng Tang, Yifan Gao, Zheng Li, Haodong Wang, Zhaoxuan Tan, Yichuan Li, Qingyu Yin, Bing Yin, and Meng Jiang. 2025. IHEval: Evaluating language models on following the instruction hierarchy. In Proceedings of the 2025 Conference of… find
  19. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/purpura-2026-deconstructing-instruction-following-new-benchmark/