Elamur Research⌕
☰
home / papers / 2026 / Concept rather than Document: Context Compression via AMR-based Conceptual Entropy

Concept rather than Document: Context Compression via AMR-based Conceptual Entropy

Paper · 2026 · Kaize Shi, Xueyao Sun, Xiaohui Tao et al.

Summary

RAG systems often pass LLMs long retrieved documents full of material unrelated to the question, which can hurt reasoning and slow inference. Existing compression methods filter on lexical or token-level signals and can break apart meaningful units. Kaize Shi and colleagues instead compress context at the level of concepts using Abstract Meaning Representation (AMR), with no supervision. Each retrieved document is parsed into sentence-level AMR graphs, and the relation edges are dropped. Each concept node gets an entropy score derived from the parser's own prediction uncertainty, combined across subword tokens. A significance test keeps only concepts with reliably high scores, and these are mapped back to their original wording in the source text. The method was tested on PopQA and EntityQuestions with ten LLM backbones, against vanilla RAG and several compression baselines. It achieved higher accuracy than these alternatives while cutting context to roughly half its original length and lowering latency. The authors present it as the first use of AMR-based conceptual entropy for context engineering.

Key points

  • Parser: AMR graphs come from an mBart-based multilingual parser (BramVanroy's multilingual-text-to-amr) trained on the AMR 3.0 corpus (LDC2020T02). The multilingual parser was chosen because web-retrieved text may contain non-English characters.
  • Controlled evaluation: only retrieved documents that contain a correct answer are kept. This isolates the effect of compression but leaves noisy, real-world retrieval untested.
  • Baselines: TF-IDF, SelCon (Selective-Context), Lingua, Keywords and Summary.
  • Backbones: ten models, labelled G-1.3, G-2.7, O-1.3, O-2.7, b-560, b-7b1, L-13, L3.1-8, DS-V2 and Q3-32.
  • Results:
    • Compressed contexts average about 50% of vanilla length with stable accuracy.
    • Keywords and Summary produce shorter contexts but lose factual cues and accuracy.
    • On long-context instances, gains are smaller but variance across backbones is notably lower.
  • Limitations: the method depends on AMR parser quality and ignores document-level structure. It also adds offline preprocessing cost for parsing and entropy scoring.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore180 works · 34,304 citations🎓Hong Kong Polytechnic UniversityHong Kong · China84 works · 4,956 citations🎓University of Technology SydneySydney · Australia18 works · 12,833 citations🎓Wuhan University of TechnologyWuhan · China4 works · 261 citations🎓University of Southern QueenslandToowoomba · Australia3 works · 413 citations

Also on the byline: Education University of Hong Kong.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Semantic parsing & meaning representation · 185 works

Cite

Cite as Kaize Shi, Xueyao Sun, Xiaohui Tao, Lin Li, Qika Lin and Guandong Xu. 2026. Concept rather than Document: Context Compression via AMR-based Conceptual Entropy. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.590

@inproceedings{shi2026concept,
  title = {{Concept rather than Document: Context Compression via AMR-based Conceptual Entropy}},
  author = {Kaize Shi, Xueyao Sun, Xiaohui Tao, Lin Li, Qika Lin and Guandong Xu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.590},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.590}
}

References

57 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. 2025. The unreasonable effectiveness of entropy minimization in llm reasoning. arXiv preprint arXiv:2505.15134. find
  2. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discou…
  3. Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021. One spring to rule them both: Symmetric amr semantic parsing and generation without a complex pipeline. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 12564–12573. find
  4. Jeffrey R Binder, Rutvik H Desai, William W Graves, and Lisa L Conant. 2009. Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral cortex, 19(12):2767–2796. find
  5. Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata. find
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  7. Qi Cao, Takeshi Kojima, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Unnatural error correction: GPT-4 can almost perfectly handle unnatural scrambled text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8898–8913, Singapore. Association for Computationa… find
  8. Huiyao Chen, Meishan Zhang, Jing Li, Min Zhang, Lilja Øvrelid, Jan Hajič, and Hao Fei. 2025. Semantic role labeling: A systematical survey. arXiv preprint arXiv:2502.08660.
  9. Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. 2024. xrag: Extreme context compression for retrieval-augmented generation with one token. Advances in Neural Information Processing Systems, 37:109487–109516. find
  10. DeepSeek-AI. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. Preprint, arXiv:2405.04434. find
  11. Changde Du, Kaicheng Fu, Bincheng Wen, Yi Sun, Jie Peng, Wei Wei, Ying Gao, Shengpei Wang, Chuncheng Zhang, Jinpeng Li, and 1 others. 2025. find
  12. Human-like object concept representations emerge naturally in multimodal large language models. Nature Machine Intelligence, pages 1–16. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal,… — The Llama 3 Herd of Models
  13. Evelina Fedorenko, Steven T Piantadosi, and Edward AF Gibson. 2024. Language is primarily a tool for communication rather than thought. Nature, 630(8017):575–586. find
  14. Tomoyasu Horikawa. 2025. Mind captioning: Evolving descriptive text of mental content from human brain activity. Science Advances, 11(45):eadw1464. find
  15. Taeho Hwang, Sukmin Cho, Soyeong Jeong, Hoyun Song, SeungYoon Han, and Jong C Park. 2024. Exit: Context-aware extractive compression for enhancing retrieval-augmented generation. arXiv preprint arXiv:2412.12559. find
  16. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research.
  17. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358–13376, Singapore. Association for Compu…
  18. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), page…
  19. Jiajie Jin, Xiaoxi Li, Guanting Dong, Yuyao Zhang, Yutao Zhu, Yongkang Wu, Zhonghua Li, Qi Ye, and Zhicheng Dou. 2025a. Hierarchical document refinement for long-context retrieval-augmented generation. arXiv preprint arXiv:2505.10413.
  20. Yiqiao Jin, Kartik Sharma, Vineeth Rakesh, Yingtong Dou, Menghai Pan, Mahashweta Das, and Srijan Kumar. 2025b. Sara: Selective and adaptive retrieval-augmented generation with context compression. arXiv preprint arXiv:2507.05633. find
  21. Zhijing Jin, Yuen Chen, Fernando Gonzalez Adauto, Jiarui Liu, Jiayi Zhang, Julian Michael, Bernhard Schölkopf, and Mona Diab. 2024. Analyzing the role of semantic representations in the era of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Associati… find
  22. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781. Asso…
  23. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations. find
  24. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, and 1 others. 2022. Bloom: A 176bparameter open-access multilingual language model. find
  25. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Igor Kulikov, Vishrav Chaudhary, Sebastian Wang, Wen-tau Yih Barta, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing System…
  26. Chenliang Li, Bin Bi, Ming Yan, Wei Wang, and Songfang Huang. 2021. Addressing semantic drift in generative question answering with auxiliary extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural… find
  27. Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023a. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353, Singapore. Association for Computational Linguistics.
  28. Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. 2023b. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  29. Fei Liu, Jeffrey Flanigan, Sam Thomson, Norman Sadeh, and Noah A. Smith. 2015. Toward abstractive summarization using semantic representations. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 107…
  30. Shengjie Liu, Jing Wu, Jingyuan Bao, Wenyi Wang, Naira Hovakimyan, and Christopher G Healey. 2024. Towards a robust retrieval-based summarization system. arXiv preprint arXiv:2403.19889. — [arXiv:2403.19889](https://arxiv.org/abs/2403.19889) find
  31. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… find
  32. Wenhao Mao, Chengbin Hou, Tianyu Zhang, Xinyu Lin, Ke Tang, and Hairong Lv. 2024. Parse trees guided llm prompt compression. arXiv preprint arXiv:2409.15395. — [arXiv:2409.15395](https://arxiv.org/abs/2409.15395) find
  33. Lingrui Mei, Jiayu Yao, Yuyao Ge, Yiwei Wang, Baolong Bi, Yujun Cai, Jiazhi Liu, Mingyu Li, Zhong-Zhi Li, Duzhen Zhang, and 1 others. 2025. A survey of context engineering for large language models. arXiv preprint arXiv:2507.13334. — [arXiv:2507.13334](https://arxiv.org/abs/2507.13334) find
  34. Dang Nguyen, Ali Payani, and Baharan Mirzasoleiman. 2025. Beyond semantic entropy: Boosting LLM uncertainty quantification with pairwise semantic similarity. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4530– 4540, Vienna, Austria. Association for Computational Lingu… find
  35. Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. 2024. Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities. In Advances in Neural Information Processing Systems, volume 37, pages 8901–8929. Curran Associates, Inc. find
  36. Stephen Robertson, Hugo Zaragoza, and 1 others. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. find
  37. Timothy T Rogers, Matthew A Lambon Ralph, Peter Garrard, Sasha Bozeat, James L McClelland, John R Hodges, and Karalyn Patterson. 2004. Structure and deterioration of semantic memory: a neuropsychological and computational investigation. Psychological review, 111(1):205. find
  38. Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple entity-centric questions challenge dense retrievers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6138–6148, Online and Punta Cana, Dominican Republic. Association for…
  39. Chunting Shi, Michihiro Yasunaga, Isabelle Augenstein, Nikos Voskarides, Mikel Artetxe, Xiang Ren, Xiaozhong Wan, Antoine Bosselut, Dragomir Radev, Wenpeng Yin, and 1 others. 2023. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652.
  40. Kaize Shi, Xueyao Sun, Qing Li, and Guandong Xu. 2024. Compressing long context for enhancing rag with amr-based concept distillation. arXiv preprint arXiv:2405.03085. — [arXiv:2405.03085](https://arxiv.org/abs/2405.03085) find
  41. Linfeng Song, Daniel Gildea, Yue Zhang, Zhiguo Wang, and Jinsong Su. 2019. Semantic neural machine translation using amr. Transactions of the Association for Computational Linguistics, 7:19–31.
  42. Siddharth Suresh, Kushin Mukherjee, Xizheng Yu, Wei-Chun Huang, Lisa Padua, and Timothy Rogers. 2023. Conceptual structure coheres in human cognition but not in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 722–738, Singapore.… find
  43. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  44. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  45. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  46. Sourav Verma. 2024. Contextual compression in retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2409.13385.
  47. Jingyao Wang, Wenwen Qiang, Zeen Song, Changwen Zheng, and Hui Xiong. 2025. Learning to think: Information-theoretic reinforcement fine-tuning for llms. arXiv preprint arXiv:2505.10425. — [arXiv:2505.10425](https://arxiv.org/abs/2505.10425) find
  48. Shira Wein and Juri Opitz. 2024. A survey of AMR applications. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6856–6875, Miami, Florida, USA. Association for Computational Linguistics. find
  49. EC Wit and Marie Gillette. 1999. What is linguistic redundancy. University of Chicago. find
  50. Kevin Wu, Eric Wu, and James Zou. 2024. Clasheval: Quantifying the tug-of-war between an LLM’s internal prior and external evidence. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  51. Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2024. RE-COMP: Improving retrieval-augmented LMs with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations. find
  52. Qihui Xu, Yingying Peng, Samuel A Nastase, Martin Chodorow, Minghua Wu, and Ping Li. 2025. Large language models without grounding recover non-sensorimotor but not sensorimotor features of human concepts. Nature human behaviour, pages 1–16. find
  53. Mingjia Yin, Chuhan Wu, Yufei Wang, Hao Wang, Wei Guo, Yasheng Wang, Yong Liu, Ruiming Tang, Defu Lian, and Enhong Chen. 2024. Entropy law: The story behind data compression and llm performance. arXiv preprint arXiv:2407.06645. find
  54. Jiahuan Zhang, Tianheng Wang, Hanqing Wu, Ziyi Huang, Yulong Wu, Dongbai Chen, Linfeng Song, Yue Zhang, Guozheng Rao, and Kaicheng Yu. 2025. Sr-llm: Rethinking the structured representation in large language model. arXiv preprint arXiv:2502.14352.
  55. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, and 1 others. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
  56. Chuyue Zhou, Wangjie You, Juntao Li, Jing Ye, Kehai Chen, and Min Zhang. 2023. INFORM : Information eNtropy based multi-step reasoning FOR large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3565–3576, Singapore. Association for Com… find
  57. Jiawei Zhou, Tahira Naseem, Ramón Fernandez Astudillo, and Radu Florian. 2021. AMR parsing with action-pointer transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5585–5598, Online. Asso… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2026-concept-rather-than-document-context/