Elamur Research⌕
☰
home / papers / 2025 / Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models

Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models

Paper · 2025 · Junfeng Tian, Da Zheng, Yang Chen et al.

Summary

Extending large language models to very long contexts is difficult for three reasons. Natural documents of 32K–128K tokens are scarce. Upsampling long data shifts the training distribution and can hurt short-context skills. Attention over long sequences is also expensive. The authors propose Untie the Knots (UtK), an augmentation applied during continued pre-training that leaves the existing data mixture unchanged.

UtK works in three steps:

  • Chunking: documents are cut at random points, and the boundaries are marked with special "knot" tokens and chunk IDs.
  • Shuffling: chunks from several documents are interleaved into one long sequence.
  • Untying: the model learns to find the matching continuation of each fragment and, in a backtracing task, to output a document's chunk IDs in the correct order.

The authors train Qwen2 models at 7B and 72B parameters on 20B tokens. UtK beats ABF, length upsampling, and training-free methods such as YaRN and DCA on 128K benchmarks. It also keeps short-context performance nearly intact and reaches strong results with far fewer tokens. The models and processing code are publicly released.

Key points

  • On RULER at 128K, UtK reaches 75% accuracy with the 7B model and 84.5% with the 72B model. The authors report gains of 15.0% on RULER and 17.2% on LV-Eval for 128K tasks, with over 90% of 32K-level performance retained.
  • On short-context benchmarks (BBH, NQ, TriviaQA, HumanEval, MBPP, GSM8K), the average changes little: −1.4% for Qwen2-UtK-7B, −0.4% for 72B, and +0.7% for Llama3.1-UtK-8B.
  • Training efficiency is high: a UtK model trained with a 192K sequence length on only 1B tokens matches an ABF baseline trained on 20B tokens.
  • In the ablations, both preserving chunk order within a document and using backtracing help. Applying UtK to 80% of sequences scores 75.0 on RULER, compared with 73.0 when order is disrupted and 74.3 without backtracing.
  • The released models are Qwen2-UtK-7B-base 128K and Qwen2-UtK-72B-base 128K (github.com/rgtjf/Untie-the-Knots). On InfiniteBench, Qwen2-UtK-ChatQA2-72B averages 47.3 on English tasks, ahead of Llama3.1-70B-Instruct at 39.8.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓East China Normal UniversityShanghai · China69 works · 2,490 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Junfeng Tian, Da Zheng, Yang Chen, Rui Wang, Colin Zhang and Debing Zhang. 2025. Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.62

@inproceedings{tian2025untie,
  title = {{Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models}},
  author = {Junfeng Tian, Da Zheng, Yang Chen, Rui Wang, Colin Zhang and Debing Zhang},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.62},
  url = {https://doi.org/10.18653/v1/2025.acl-long.62}
}

References

2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219.
  2. Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. 2024. Training-free long-context scaling of large language models. Preprint, arXiv:2402.17463. *Figure 7: Attention map of Qwen2-UtK-base 7B.* Anthropic. 2023. Model card and evaluations for claude models. \#\#… — [arXiv:2402.17463](https://arxiv.org/abs/2402.17463) find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tian-2025-untie-knots-efficient-data-augmentation/