Elamur Research⌕
☰
home / papers / 2026 / GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment

GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment

Paper · 2026 · Jiwei Tang, Zhicheng Zhang, Shunlong Wu et al.

Summary

Soft prompt compression shrinks long LLM inputs into a few learned vectors. The authors identify two weaknesses in existing methods. First, appended summary tokens tend to be dominated by a few anchor tokens during autoencoder pretraining. Second, high-level summary vectors are fed to the decoder as though they were ordinary low-level input embeddings.

GMSA is an encoder–decoder framework that addresses both problems:

  • Group Merging splits the encoder outputs into equal-sized groups and average-pools each one, so information is aggregated evenly.
  • Layer Semantic Alignment (LSA) is a small Transformer block initialized from the decoder's lowest layers. It maps the summary vectors into the decoder's input space.

Training has two stages. Autoencoder pretraining updates the LoRA-tuned encoder and LSA. Task fine-tuning then updates only the decoder. The backbones are Qwen3-4B-Instruct and LLaMA-3.2-3B.

GMSA reconstructs context better than the standard soft-compression paradigm. It beats baselines such as ICAE, Provence, LongLLMLingua and Activation Beacon on long-context QA and summarization, while keeping end-to-end latency low. This makes it a practical option for cheaper long-context and RAG inference.

Key points

  • QA results with Qwen3-4B-Instruct (EM). Benchmarks are NaturalQuestions, 2WikiMQA, HotpotQA and NarrativeQA.
    • At 4x compression, GMSA reaches 60.38 / 55.75 / 52.28 / 12.97, versus 43.69 / 46.56 / 42.12 / 14.38 for fine-tuned Provence.
    • NarrativeQA is the one benchmark where Provence scores higher.
  • 8x compression. GMSA scores 50.02 EM on NaturalQuestions, compared with 39.77 for Provence (FT).
  • Ablation (NaturalQuestions EM, 4x, default 60.38).
    • Without autoencoder pretraining: 48.81.
    • Without Group Merging: 21.88.
    • Without LSA: 20.15.
    • With randomly initialized LSA: 19.47.
  • Stress test. On NaturalQuestions with LLaMA-3.2-3B, rates from 2x to 32x were tested. GMSA performed best at every ratio, while LongLLMLingua and Activation Beacon declined sharply at high compression.
  • Limitation. The two-stage training adds some overhead compared with training-free compression methods. A single LSA layer is reported to be sufficient.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Seoul National UniversitySeoul · South Korea41 works · 45,671 citations🎓Sun Yat-sen UniversityGuangzhou · China53 works · 9,185 citations🔬Peng Cheng LaboratoryShenzhen · China38 works · 2,504 citations🎓Tsinghua Shenzhen International Graduate SchoolShenzhen · China28 works · 3,035 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Jiwei Tang, Zhicheng Zhang, Shunlong Wu, Jingheng Ye, Lichen Bai, Zitai Wang, Tingwei Lu, Lin Hai, Yiming Zhao, Hai-Tao Zheng and Hong-Gee Kim. 2026. GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1324

@inproceedings{tang2026gmsa,
  title = {{GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment}},
  author = {Jiwei Tang, Zhicheng Zhang, Shunlong Wu, Jingheng Ye, Lichen Bai, Zitai Wang, Tingwei Lu, Lin Hai, Yiming Zhao, Hai-Tao Zheng and Hong-Gee Kim},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1324},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1324}
}

References

51 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro…
  2. Yun-Hao Cao, Yangsong Wang, Shuzheng Hao, Zhenxing Li, Chengjun Zhan, Sichao Liu, and Yi-Qi Hu. 2025. Efpc: Towards efficient and flexible prompt compression. Preprint, arXiv:2503.07956. — [arXiv:2503.07956](https://arxiv.org/abs/2503.07956) find
  3. Lizhe Chen, Binjia Zhou, Yuyao Ge, Jiayi Chen, and Shiguang Ni. 2025. Pis: Linking importance sampling and attention mechanisms for efficient prompt compression. arXiv preprint arXiv:2504.16574. — [arXiv:2504.16574](https://arxiv.org/abs/2504.16574) find
  4. Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. 2024. xRAG: Extreme context compression for retrieval-augmented generation with one token. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  5. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023. Adapting language models to compress contexts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3829–3846, Singapore. Association for Computational Linguistics.
  6. Nadezhda Chirkova, Thibault Formal, Vassilina Nikoulina, and Stéphane Clinchant. 2025. Provence: efficient and robust context pruning for retrieval-augmented generation. Preprint, arXiv:2501.16214. — [arXiv:2501.16214](https://arxiv.org/abs/2501.16214) find
  7. Yuhong Dai, Jianxun Lian, Yitian Huang, Wei Zhang, Mingyang Zhou, Mingqi Wu, Xing Xie, and Hao Liao. 2025. Pretraining context compressor for large language models with embedding-based memory. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  8. Chenlong Deng, Zhisong Zhang, Kelong Mao, Shuaiyi Li, Tianqing Fang, Hongming Zhang, Haitao Mi, Dong Yu, and Zhicheng Dou. 2025. Unigist: Towards general and hardware-aligned sequence-level long context compression. CoRR, abs/2509.15763. find
  9. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find
  10. Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In ACL (1), pages 1074– 1084. Association for Computational Linguistics.
  11. Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2024. In-context autoencoder for context compression in a large language model. In The Twelfth International Conference on Learning Representations. find
  12. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Comm…
  13. Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vis… find
  14. Taeho Hwang, Sukmin Cho, Soyeong Jeong, Hoyun Song, SeungYoon Han, and Jong C. Park. 2025. Exit: Context-aware extractive compression for enhancing retrieval-augmented generation. Preprint, arXiv:2412.12559. find
  15. Huiqiang Jiang, Qianhui Wu, , Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1…
  16. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358–13376, Singapore. Association for Compu…
  17. Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, and Yongfeng Zhang. 2025. Exploring concept depth: How large language models acquire knowledge and concept at different layers? In Proceedings of the…
  18. Tomás Kociský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Trans. Assoc. Comput. Linguistics, 6:317–328.
  19. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–…
  20. Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353, Singapore. Association for Computational Linguistics.
  21. Zongqian Li, Yixuan Su, and Nigel Collier. 2024. 500xcompressor: Generalized prompt compression for large language models. Preprint, arXiv:2408.03094.
  22. Zongqian Li, Yixuan Su, and Nigel Collier. 2025. 500xcompressor: Generalized prompt compression for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25081–25091.
  23. Huanxuan Liao, Wen Hu, Yao Xu, Shizhu He, Jun Zhao, and Kang Liu. 2025. Beyond hard and soft: Hybrid context compression for balancing local and global information retention. Preprint, arXiv:2505.15774. — [arXiv:2505.15774](https://arxiv.org/abs/2505.15774) find
  24. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  25. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
  26. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  27. Zhu Liu, Cunliang Kong, Ying Liu, and Maosong Sun. 2024b. Fantastic semantics and where to find them: Investigating which layers of generative LLMs reflect lexical semantics. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14551–14558, Bangkok, Thailand. Association for… find
  28. Jesse Mu, Xiang Lisa Li, and Noah Goodman. 2023. Learning to compress prompts with gist tokens. In Thirty-seventh Conference on Neural Information Processing Systems.
  29. Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Ruhle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Findings of the Associat…
  30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  31. Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. CoRR, abs/2505.06708. — [arXiv:2505.06708](https://arxiv.org/abs/2505.06708) find
  32. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  33. David Rau, Shuai Wang, Hervé Déjean, Stéphane Clinchant, and Jaap Kamps. 2025. Context embeddings for efficient answer generation in retrieval-augmented generation. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, WSDM ’25, page 493–502, New York, NY, USA.… find
  34. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In ICML, Proceedings of Machine Learning Research, pages 31210–31227. PMLR.
  35. Jiwei Tang, Jin Xu, Tingwei Lu, Zhicheng Zhang, YimingZhao YimingZhao, LinHai LinHai, and Hai-Tao Zheng. 2025. Perception compressor: A training-free prompt compression framework in long context scenarios. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4093–4108, Alb… find
  36. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, and 1 others. 2025. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534.
  37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  38. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv. find
  39. Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2024. Recomp: Improving retrieval-augmented lms with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations. find
  40. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
  41. Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, and Yansong Tang. 2024. Voco-llama: Towards vision compression with large language models. arXiv preprint arXiv:2406.12275. — [arXiv:2406.12275](https://arxiv.org/abs/2406.12275) find
  42. Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. 2024. CompAct: Compressing retrieved documents actively for question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21424–21439, Miami, Florida, USA. Association for…
  43. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, and 1 others. 2025. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471.
  44. Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. 2024. Long context compression with activation beacon. arXiv preprint arXiv:2401.03462.
  45. Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. 2025. Long context compression with activation beacon. In ICLR. Open-Review.net.
  46. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  47. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Informati… find
  48. Yiming Zhao, Jiwei Tang, Shimin Di, Libin Zheng, Jianxing Yu, and Jian Yin. 2025a. Cos: Towards optimal event scheduling via chain-of-scheduling. Preprint, arXiv:2511.12913. — [arXiv:2511.12913](https://arxiv.org/abs/2511.12913) find
  49. Yunlong Zhao, Haoran Wu, and Bo Xu. 2025b. Leveraging attention to effectively compress prompts for long-context llms. Proceedings of the AAAI Conference on Artificial Intelligence, 39(24):26048–26056. find
  50. Fengwei Zhou, Jiafei Song, Wenjin Jason Li, Gengjian Xue, Zhikang Zhao, Yichao Lu, and Bailin Na. 2025. Mooscomp: Improving lightweight long-context compressor via mitigating over-smoothing and incorporating outlier scores. arXiv preprint arXiv:2504.16786. — [arXiv:2504.16786](https://arxiv.org/abs/2504.16786) find
  51. Impact of Different Number of LSA Layers find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2026-gmsa-enhancing-context-compression-group/