Elamur Research⌕
☰
home / papers / 2026 / Glyph: Scaling Context Windows via Visual-Text Compression

Glyph: Scaling Context Windows via Visual-Text Compression

Paper · 2026 · Jiale Cheng, Yusen Liu, Xinyu Zhang et al.

Summary

Long-context LLMs usually scale by handling more text tokens, which raises compute and memory costs. Glyph takes a different route. Instead of enlarging the token window, it packs more source text into each token: documents are rendered as images and fed to a vision-language model. Built on GLM-4.1V-9B-Base, the pipeline has three stages:

  • continual pre-training on rendered long texts in varied visual styles;
  • an LLM-guided genetic search that tunes rendering parameters (font, size, DPI, layout) to balance compression against accuracy;
  • post-training via SFT and RL, with an extra OCR objective.

On benchmarks such as LongBench and MRCR, Glyph shrinks input roughly three to four times. It stays competitive with similarly sized text-only models like Qwen3-8B and speeds up both inference and fine-tuning. At 8× compression, a 128K-window model handles inputs equivalent to about 1M text tokens. The work matters because it adds a scaling axis independent of longer windows, sparse attention or retrieval.

Key points

  • Efficiency on 128K-token inputs: 3–4× token compression, with prefilling accelerated by as much as 4.8×, decoding sped up 4.4× and SFT running about 2× quicker than with the text backbone.
  • Illustrative case: Jane Eyre (~240K text tokens) fits into roughly 80K visual tokens, so a 128K-context VLM can read the whole novel.
  • Ablations:
    • The searched rendering configuration outperforms random and hand-designed settings at similar compression ratios.
    • OCR auxiliary tasks in SFT and RL give consistent gains.
  • Extreme 8× compression: on MRCR from 128K to 1024K tokens, Glyph is on par with GLM-4-9B-Chat-1M and Qwen2.5-1M. Training also improves real document understanding on MMLongBench-Doc (130 long PDFs).
  • Limitations and release:
    • Performance is sensitive to rendering choices.
    • Rare strings such as UUIDs in Ruler still cause recognition errors.
    • Code and model are available at github.com/thu-coai/Glyph.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🏢Zhipu AIBeijing · China22 works · 2,364 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei, Wenyi Hong, Ruiliang Lyu, Weihan Wang, Zhe Su, Xiaotao Gu, Xiao Liu, Yushi Bai, Jie Tang, Hongning Wang and Minlie Huang. 2026. Glyph: Scaling Context Windows via Visual-Text Compression. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1722

@inproceedings{cheng2026glyph,
  title = {{Glyph: Scaling Context Windows via Visual-Text Compression}},
  author = {Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei, Wenyi Hong, Ruiliang Lyu, Weihan Wang, Zhe Su, Xiaotao Gu, Xiao Liu, Yushi Bai, Jie Tang, Hongning Wang and Minlie Huang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1722},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1722}
}

References

37 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  2. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, and 1 others. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin…
  3. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. In Proceedings of ACL.
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  5. Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, and 1 others. 2025a. Minimax-m1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585.
  6. Xi Chen, Xiao Wang, Soravit Changpinyo, Anthony J Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, and 1 others. 2022. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794.
  7. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024. Longlora: Efficient fine-tuning of long-context large language models. In The International Conference on Learning Representations (ICLR).
  8. Yutao Chen, Yiren Wang, and 1 others. 2025b. Cope: Complex positional encoding for long context extrapolation. arXiv preprint arXiv:2508.18308. — [arXiv:2508.18308](https://arxiv.org/abs/2508.18308) find
  9. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and 1 others. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
  10. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  11. Team GLM, :, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, and 38 others. 2024. Chatglm: A family of large language models from glm-130b to glm-4…
  12. Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, and 1 others. 2024a. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408.16500.
  13. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and 1 others. 2024b. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290.
  14. Yunpeng Huang, Jingwei Xu, Junyu Lai, Zixu Jiang, Taolue Chen, Zenan Li, Yuan Yao, Xiaoxing Ma, Lijuan Yang, Hao Chen, and 1 others. 2023. Advancing transformer architecture in long-context large language models: A comprehensive survey. arXiv preprint arXiv:2311.12351. — [arXiv:2311.12351](https://arxiv.org/abs/2311.12351) find
  15. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  16. Philippe Laban, Alexander Richard Fabbri, Caiming Xiong, and Chien-Sheng Wu. 2024. Summary of a haystack: A challenge to long-context llms and rag systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9885–9903.
  17. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024a. Llava-next: Improved reasoning, ocr, and world knowledge. find
  18. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  19. Yang Liu, Wei Gao, and 1 others. 2024b. Long context is not long at all: A prospector of long-dependency data for large language models. arXiv preprint arXiv:2407.11234. find
  20. Baolin Peng, Zhuohan Li, and 1 others. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071. find
  21. Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, and 1 others. 2025. Rwkv-7" goose" with expressive dynamic state evolution. arXiv preprint arXiv:2503.14456.
  22. Ofir Press, Noah A. Smith, and Mike Levy. 2021. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409. find
  23. Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864.
  24. Zhen Sun, Peng Cheng, Wei He, and 1 others. 2022. Xpos: Improving position interpolation with extrapolation. arXiv preprint arXiv:2212.10554. — [arXiv:2212.10554](https://arxiv.org/abs/2212.10554) find
  25. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  26. Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, and 1 others. 2024. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640.
  27. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, and 1 others. 2024a. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37:121475–121499.
  28. Yizhi Wang, Fan Yang, and 1 others. 2024b. Longrecipe: Recipe for efficient long context generalization in large language models. arXiv preprint arXiv:2406.12345.
  29. Haoran Wei, Yaofeng Sun, and Yukun Li. 2025. Deepseek-ocr: Contexts optical compression. arXiv preprint arXiv:2510.18234.
  30. Yingsheng Wu, Yuxuan Gu, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, and Bing Qin. 2024. Extending context window of large language models from a distributional perspective. arXiv preprint arXiv:2410.01490. find
  31. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  32. Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. 2024. Gated linear attention transformers with hardware-efficient training. In International Conference on Machine Learning, pages 56501–56523. PMLR. find
  33. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of NAACL.
  34. Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, and 1 others. 2025a. Memagent: Reshaping long-context llm with multi-conv rl-based memory agent. arXiv preprint arXiv:2507.02259. — [arXiv:2507.02259](https://arxiv.org/abs/2507.02259) find
  35. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2025b. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476.
  36. Tianle Zhang, Zhuohan Li, and 1 others. 2024. Longalign: Instruction-tuning long-context llms. arXiv preprint arXiv:2401.10968. — [arXiv:2401.10968](https://arxiv.org/abs/2401.10968) find
  37. Wei Zhu, Ziheng Wang, and 1 others. 2024. Data-adaptive positional encoding for length generalization. NeurIPS. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cheng-2026-glyph-scaling-context-windows-visual/