Elamur Research⌕
☰
home / papers / 2025 / Following Length Constraints in Instructions

Following Length Constraints in Instructions

Paper · 2025 · Weizhe Yuan, Ilia Kulikov, Ping Yu et al.

Summary

Instruction-tuned LLMs and their evaluators tend to favor longer answers. Many prompts do not say how long the answer should be, which makes evaluation ambiguous and lets training exploit this length bias. Yuan et al. (Meta FAIR / NYU) suggest stating the desired length explicitly in the prompt as a word limit. They introduce two benchmarks, AlpacaEval-LI and MT-Bench-LI, which add a per-prompt maximum word count to existing AlpacaEval and MT-Bench instructions. Any answer that exceeds the limit automatically loses against a baseline that always satisfies it. Strong models such as GPT-4 Turbo break these limits surprisingly often.

To fix this, the authors propose Length-Instruction Fine-Tuning (LIFT). It augments a preference dataset with length instructions, so that a preference pair can flip when the originally preferred answer is too long. Models are then trained with DPO on both the original and augmented data. LIFT-DPO versions of Llama 2 and Llama 3 sharply reduce violations and raise win rates, with no loss on standard benchmarks. This gives a less gameable way to compare models and more user control over output length.

Key points

  • Benchmark construction:
    • AlpacaEval-LI covers 802 AlpacaEval prompts. Three prompts that already had explicit length constraints were dropped.
    • MT-Bench-LI has 240 first-turn prompts, with three length limits sampled per prompt.
    • Each word limit equals the shortest response produced by GPT-4 Turbo (11/06), Claude 3 Opus, and Mistral Large.
  • SOTA models struggle with limits: GPT-4 Turbo (2024-04-09) violated them in 49.3% of cases on AlpacaEval-LI and 44.2% on MT-Bench-LI.
  • LIFT data:
    • The base set is 3,200 OpenAssistant preference pairs.
    • Pairs whose two responses differ in length by fewer than T = 10 words are filtered out.
    • The remaining pairs are augmented into 5,954 length-instructed pairs.
  • Results on AlpacaEval-LI:
    • For Llama2-70B-Base, LIFT-DPO cut violations from 65.8% (standard DPO) to 7.1%, and win rate rose from 4.6% to 13.6%.
    • For Llama2-70B-Chat, violations fell from 15.1% to 2.7%.
    • LIFT-DPO beat R-DPO (length-regularized DPO).
    • Violations stayed below 10% even when limits were scaled down to 0.1 of their original value.
  • Further findings:
    • Length-Controlled AlpacaEval winrate can still be gamed: varying the requested length moved it from 23% to 29%.
    • LIFT-DPO also lowered violations on IFEval's upper-bound word and sentence constraints, which appear in 44 of its 541 instructions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States201 works · 100,180 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Weizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason E Weston and Jing Xu. 2025. Following Length Constraints in Instructions. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1233

@inproceedings{yuan2025following,
  title = {{Following Length Constraints in Instructions}},
  author = {Weizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason E Weston and Jing Xu},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1233},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1233}
}

References

16 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217. find
  3. Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprint arXiv:2402.07319. find
  4. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024a. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. find
  5. Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2024b. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36.
  6. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. 2023. OpenAssistant conversations–democratizing large language model alignment. arXiv preprint arXiv:2304.07327.
  7. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. 2024. Self-alignment with instruction back-translation. In The Twelfth International Conference on Learning Representations. find
  8. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. ↗
  9. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  10. Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159.
  11. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems.
  12. Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. arXiv preprint arXiv:2310.05199. find
  13. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716.
  14. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. from nltk . tokenize import word_tokenize 2 i…
  15. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing…
  16. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yuan-2025-following-length-constraints-instructions/