Elamur Research⌕
☰
home / papers / 2023 / Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering

Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering

Paper · 2023 · Yucheng Li

Summary

Transformer-based LLMs can only attend to a fixed window of tokens, and enlarging that window is costly because attention scales quadratically. This makes long documents and lengthy dialogues hard to handle. The paper proposes Selective Context, which shrinks the input before it reaches the LLM. A smaller causal language model (e.g., GPT-2, OPT, LLaMA) scores every token by self-information, its negative log-probability given the preceding text. Because self-information is additive, these scores are summed into larger units: tokens, noun phrases (merged with spaCy) or sentences. Units falling below a chosen percentile of the score distribution are dropped. The method was tested on recent arXiv papers, BBC News articles and ShareGPT conversations, all collected after ChatGPT's knowledge cutoff to limit contamination. Tasks were reconstruction, summarisation, QA and conversation, scored with BLEU, ROUGE-1 and BERTScore. Removing roughly 20–35% of the content causes only small quality losses and beats random deletion. This offers a cheap, model-agnostic way to stretch limited context budgets.

Key points

  • Datasets: Arxiv has 408 documents (only the first two sections are used), BBC News has 294, and ShareGPT has 470 conversations. Each document averages about 690–865 tokens.
  • Example compression: Filtering at the phrase level with the percentile set to 50 kept about 57% of the tokens, saving roughly 43% of the context.
  • Small losses at 20% reduction: Compared with answers from the full context, BLEU fell by only 0.05 and ROUGE-1 by 0.03. Selective Context also outperformed a random-deletion baseline at 35% reduction (about 0.3 BLEU and over 0.55 ROUGE-1).
  • Task sensitivity: Reconstruction reached ROUGE-1 of about 0.65 and BERTScore above 0.9 at reduction ratios of 0.2–0.35, but collapsed at 0.8 (BLEU 0.03). Summarisation and conversation held up well, while QA and reconstruction were more sensitive because they need fine details.
  • Code: Available at github.com/liyucheng09/Selective_Context.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yucheng Li. 2023. Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering. arXiv:2304.12102.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.12102 · arXiv 2304.12102

@misc{li2023unlocking,
  title = {{Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering}},
  author = {Yucheng Li},
  year = {2023},
  doi = {10.48550/arxiv.2304.12102},
  eprint = {2304.12102},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.12102}
}

References

8 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Banerjee, S. and Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65–72, 2005.
  2. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020.
  3. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. \#\#\# 5.4. Case Study Dong, Z., Tang, T., Li, L., and Zhao, W…
  4. To have a more straightforward impression of how Selective Context reduce context cost, we present several cases in the Appendix. Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74–81, 2004. \#\# 6. Conclusion Papineni, K., Roukos, S., Ward, T…
  5. In this paper, we introduced Selective Context to maximise the utility of fixed context length in LLMs. We demonstrated the effectiveness of our method by filtering out less informative content, providing a more compact and efficient context representation for LLMs without sacrificing their Shannon,… find
  6. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  7. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  8. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. *Figure 6. Selective Context with reduction ratio of 0.5 on conversation transcripts.* find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2023-unlocking-context-constraints-llms-enhancing/