Elamur Research⌕
☰
home / papers / 2026 / IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking

IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking

Paper · 2026 · Zechen Sun, Yuyang Sun, Zecheng Tang et al.

Summary

The paper tackles a weakness of LLMs in open-ended long-form writing: once the requested length goes beyond roughly 2,000 words, both quality and adherence to the target length drop sharply. The authors call this "Length Collapse." Their preliminary studies show that reasoning-enhanced models such as DeepSeek-R1 do better than standard ones. However, the up-front outline these models produce loses its usefulness as the text grows. To address this, the authors propose IS-CoT, a framework in which the model itself alternates between four steps throughout generation: a global plan, local step plans, writing content, and reflecting on progress (including word counts). No external agent pipeline is needed. They build about 5,000 training examples with such interleaved traces using a three-stage, multi-teacher pipeline, then fine-tune Qwen3-8B into IS-Writer-8B. The resulting small model beats much larger open and proprietary systems on long-form benchmarks, and it holds up especially well at very long lengths. This suggests that ongoing planning matters more than model scale for controllable long text.

Key points

  • Preliminary study: 30 bilingual prompts (English/Chinese) at five target lengths, from 500 to 32,000 words, were run on GPT-4o, LLaMA-3.3-70B-Instruct, DeepSeek-V3 and DeepSeek-R1. The average Length Following score fell to about 15 in the 2k–8k word range and below 10 at 16k–32k.
  • Reasoning vs. standard models: On LongBench-Write, the "Thinking" variants (Qwen3-8B, R1-Distill-LLaMA-8B, GLM-Z1-9B) beat their instruct counterparts. They still degraded as target length increased.
  • Data pipeline: The three stages are instruction refinement, interleaved synthesis using DeepSeek and Qwen teachers, and quality filtering (length score above 90). Training sequences use tagged segments for global_plan, step_plan, content and reflection.
  • Results: IS-Writer-8B averages 88.25 on LongBench-Write and 8.60 on WritingBench. It scores +3.08 over DeepSeek-V3.2, +4.58 over Gemini-2.5-Flash and +10.28 over other writing-enhanced LLMs, with notable robustness in the [4k, 20k) range.
  • Limitations: The approach depends on the quality of the teacher models. The extra planning and reflection tokens also increase inference cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Soochow UniversitySuzhou · China63 works · 4,507 citations🔬PLA Academy of Military ScienceBeijing · China3 works · 510 citations🎓Soochow UniversityTaipei · Taiwan2 works · 26 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Zechen Sun, Yuyang Sun, Zecheng Tang, Juntao Li, Wenpeng Hu, Wenliang Chen, Zhunchen Luo, Guotong Geng and Min Zhang. 2026. IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.911

@inproceedings{sun2026is,
  title = {{IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking}},
  author = {Zechen Sun, Yuyang Sun, Zecheng Tang, Juntao Li, Wenpeng Hu, Wenliang Chen, Zhunchen Luo, Guotong Geng and Min Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.911},
  url = {https://doi.org/10.18653/v1/2026.acl-long.911}
}

References

41 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2024. Make your llm fully utilize the context. Advances in Neural Information Processing Systems, 37:62160–62188. find
  2. Anthropic. 2024. Introducing claude 3.5 sonnet. https://www.anthropic.com/research/ claude-3-5-sonnet. ↗
  3. Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025. Longwriter: Unleashing 10,000+ word generation from long context llms. In The Thirteenth International Conference on Learning Representations.
  4. George Arthur Baker, Ankush Raut, Sagi Shaier, Lawrence E Hunter, and Katharina von der Wense. 2024. Lost in the middle, and in-between: Enhancing language models’ ability to reason over long contexts in multi-hop qa. arXiv preprint arXiv:2412.10079. — [arXiv:2412.10079](https://arxiv.org/abs/2412.10079) find
  5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  6. Chengwei Dai, Kun Li, Wei Zhou, and Songlin Hu. 2024. Beyond imitation: Learning key reasoning steps from dual chain-of-thoughts in reasoning distillation. arXiv preprint arXiv:2405.19737. — [arXiv:2405.19737](https://arxiv.org/abs/2405.19737) find
  7. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, and 245 others. 2025. Deepseek-v3.2: Pushing the frontier of open…
  8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  9. Kaituo Feng, Changsheng Li, Xiaolu Zhang, Jun Zhou, Ye Yuan, and Guoren Wang. 2024. Keypoint-based progressive chain-of-thought distillation for llms. In Proceedings of the 41st International Conference on Machine Learning, pages 13241–13255. find
  10. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, and 1 others. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793.
  11. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others. 2025. Deepseekr1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638. find
  12. Alexander Gurung and Mirella Lapata. 2025. Learning to reason for long-form story generation. In Second Conference on Language Modeling. find
  13. Jacqueline He, Howard Yen, Margaret Li, Shuyue Stella Li, Zhiyuan Zeng, Weijia Shi, Yulia Tsvetkov, Danqi Chen, Pang Wei Koh, and Luke Zettlemoyer. 2025. find
  14. Precise information control in long-form text generation. In Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS 2025). find
  15. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computati…
  16. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  17. Xuanyu Lei, Chenliang Li, Yuning Wu, Kaiming Liu, Weizhou Shen, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu. 2025. Writing-rl: Advancing long-form writing via adaptive curriculum reinforcement learning. arXiv preprint arXiv:2506.05760.
  18. Yi Liang, You Wu, Honglei Zhuang, Li Chen, Jiaming Shen, Yiling Jia, Zhen Qin, Sumit Sanghai, Xuanhui Wang, Carl Yang, and 1 others. 2024. Integrating planning into single-turn long-form text generation. arXiv preprint arXiv:2410.06203. — [arXiv:2410.06203](https://arxiv.org/abs/2410.06203) find
  19. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  20. Xin Liu, Lechen Zhang, Sheza Munir, Yiyang Gu, and Lu Wang. 2025. Verifact: Enhancing long-form factuality evaluation with refined fact extraction and reference facts. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17919–17936.
  21. Chau Minh Pham, Simeng Sun, and Mohit Iyyer. 2024. Suri: Multi-constraint instruction following in long-form text generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1722–1753, Miami, Florida, USA. Association for Computational Linguistics.
  22. Bowen Ping, Jiali Zeng, Fandong Meng, Shuo Wang, Jie Zhou, and Shanghang Zhang. 2025. LongDPO: Unlock better long-form generation abilities for LLMs via critique-augmented stepwise information. In Findings of the Association for Computational Linguistics: ACL 2025, pages 7613–7632, Vienna, Austria.… find
  23. Haoran Que and Wenge Rong. 2025. Pic: Unlocking long-form text generation capabilities of large language models via position id compression. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6982–6995. find
  24. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  25. Haozhe Wang, Haoran Que, Qixin Xu, Minghao Liu, Wangchunshu Zhou, Jiazhan Feng, Wanjun Zhong, Wei Ye, Tong Yang, Wenhao Huang, and 1 others. 2025a. Reverse-engineered reasoning for open-ended generation. arXiv preprint arXiv:2509.06160. — [arXiv:2509.06160](https://arxiv.org/abs/2509.06160) find
  26. Qianyue Wang, Jinwu Hu, Zhengping Li, Yufeng Wang, Daiyuan Li, Yu Hu, and Mingkui Tan. 2025b. Generating long-form story using dynamic hierarchical outlining with memory-enhancement. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lin… find
  27. Ruida Wang, Yuxin Li, Yi R. Fung, and Tong Zhang. 2025c. Let’s reason formally: Natural-formal hybrid reasoning enhances LLM’s math capability. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 16794–16820, Suzhou, China. Association for Computational L…
  28. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
  29. Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Qingsong Wen, Wei Ye, and 1 others. 2024. Autosurvey: Large language models can automatically write surveys. Advances in neural information processing systems, 37:115119–115145. find
  30. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  31. Yuhao Wu, Yushi Bai, Zhiqiang Hu, Roy Ka-Wei Lee, and Juanzi Li. 2025a. Longwriter-zero: Mastering ultra-long text generation via reinforcement learning. arXiv preprint arXiv:2506.18841. — [arXiv:2506.18841](https://arxiv.org/abs/2506.18841) find
  32. Yuhao Wu, Yushi Bai, Zhiqiang Hu, Juanzi Li, and Roy Ka-Wei Lee. 2025b. Superwriter: Reflection-driven long-form generation with large language models. arXiv preprint arXiv:2506.04180.
  33. Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang. 2025c. Writingbench: A comprehensive benchmark for generative writing. Preprint, arXiv:2503.05244. — [arXiv:2503.05244](https://arxiv.org/abs/2503.05244) find
  34. Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Jiaming Ji, Yingying Zhang, and 1 others. 2025. Redstar: Does scaling long-cot data unlock better slow-reasoning systems? arXiv preprint arXiv:2501.11284. find
  35. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388. scripts) makes up the largest part (48.4%), helping the model learn narrative structures. Moreover, pro…
  36. Cehao Yang, Xueyuan Lin, Chengjin Xu, Xuhui Jiang, Shengjie Ma, Aofan Liu, Hui Xiong, and Jian Guo. 2025b. Longfaith: Enhancing long-context reasoning in llms with faithful synthetic data. arXiv preprint arXiv:2502.12583. find
  37. Ruihan Yang, Caiqi Zhang, Zhisong Zhang, Xinting Huang, Dong Yu, Nigel Collier, and Deqing Yang. 2025c. Uncle: Uncertainty expressions in long-form generation. arXiv preprint arXiv:2505.16922. — [arXiv:2505.16922](https://arxiv.org/abs/2505.16922) find
  38. Wen Yang, Minpeng Liao, and Kai Fan. 2025d. Markov chain of thought for efficient mathematical reasoning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7132– 71… find
  39. Huifeng Yin, Yu Zhao, Minghao Wu, Xuanfan Ni, Bo Zeng, Tianqi Shi, Liangying Shao, Chenyang Lyu, Longyue Wang, Weihua Luo, and 1 others. 2025. Marco-o1 v2: Towards widening the distillation bottleneck for reasoning models. In Proceedings of the 63rd Annual Meeting of the Association for Computationa…
  40. Xueguan Zhao, Wenpeng Lu, Chaoqun Zheng, Weiyu Zhang, Jiasheng Si, and Deyu Zhou. 2025. Plan dynamically, express rhetorically: A debate-driven rhetorical framework for argumentative writing. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9562–9584. find
  41. Xianwei Zhuang, Zhihong Zhu, Zhichang Wang, Xuxin Cheng, and Yuexian Zou. 2025. Unicott: A unified framework for structural chain-of-thought distillation. In The Thirteenth International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-is-cot-breaking-long-form/