Elamur Research
☰
home / papers / 2022 / Large Language Models Struggle to Learn Long-Tail Knowledge

Large Language Models Struggle to Learn Long-Tail Knowledge

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2211.08411

Summary

This paper asks whether the factual knowledge of large language models depends on how often a fact appears in their pre-training data. The authors ran the DBpedia Spotlight Entity Linker over several large corpora: The Pile, ROOTS (En), C4, OpenWebText and a 2018 Wikipedia dump. For each question in Natural Questions and TriviaQA, they took the main question entity and answer entity and counted the documents where both occur. They then compared these counts with few-shot closed-book QA accuracy for model families such as BLOOM and GPT-Neo.

Accuracy rises sharply with the number of relevant documents, and the pattern holds across datasets, corpora and model sizes. A re-training experiment that removes relevant documents suggests the link is causal. Bigger models do handle rare facts better, but extrapolating the trend implies an impractically large scale-up would be needed. Retrieval augmentation largely removes the dependence on frequency. The work matters because it shows that LLM factual competence is concentrated in well-represented knowledge and points to retrieval as a better route to rare facts.

Key points

  • Entity linking pipeline. Linking 2.1TB of pre-training text took about three weeks on a 128-core CPU machine. The code and entity data are public at github.com/nkandpa2/long_tail_knowledge.
  • Human check of the pipeline. On 300 TriviaQA pairs, 33% of sampled documents fully answered the question and another 27% were partially relevant, for roughly 60% precision.
  • Frequency effect. BLOOM-176B goes from about 25% to over 55% accuracy as relevant document counts grow from 10¹ to 10⁴.
  • Causal test. A 4.8B-parameter model re-trained without certain relevant documents lost significant accuracy on the affected questions.
  • Scaling and retrieval. Accuracy grows log-linearly with parameter count, and competitive long-tail QA would require roughly a quadrillion parameters. Adding three BM25 paragraphs to GPT-Neo on Natural Questions removes most of the low accuracy on rare questions, although BM25 recall itself depends mildly on document counts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

258 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 247 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2211.08411 · arXiv 2211.08411

@misc{kandpal2022large,
  title = {{Large Language Models Struggle to Learn Long-Tail Knowledge}},
  author = {Kandpal, Nikhil and Deng, Haikang and Roberts, Adam and Wallace, Eric and Raffel, Colin},
  year = {2022},
  doi = {10.48550/arxiv.2211.08411},
  eprint = {2211.08411},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2211.08411}
}

References

37 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akyürek, E., Bolukbasi, T., Liu, F., Xiong, B., Tenney, I., Andreas, J., and Guu, K. Tracing knowledge in language models back to the training data. In Findings of EMNLP, 2022. find
  2. Han, X. and Tsvetkov, Y. ORCA: Interpreting prompted language models via locating supporting data evidence in the ocean of pretraining data. arXiv preprint arXiv:2205.12600, 2022. — [arXiv:2205.12600](https://arxiv.org/abs/2205.12600) find
  3. Black, S., Leo, G., Wang, P., Leahy, C., and Biderman, S. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, 2021. find
  4. Ippolito, D., Tramèr, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., Choquette-Choo, C. A., and Carlini, N. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546, 2022. — [arXiv:2210.17546](https://arxiv.org/abs/2210.17546) find
  5. Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. GPT-Neox-20B: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022. find our card
  6. Izacard, G. and Grave, E. Distilling knowledge from reader to retriever for question answering. In ICLR, 2021. find
  7. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In NeurIPS, 2020. find
  8. Jagielski, M., Thakkar, O., Tramer, F., Ippolito, D., Lee, K., Carlini, N., Wallace, E., Song, S., Thakurta, A., Papernot, N., et al. Measuring forgetting of memorized training examples. In ICLR, 2023. find
  9. Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium, 2019. find our card
  10. Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017. find our card
  11. Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A., and Raffel, C. Extracting training data from large language models. In USENIX Security Symposium, 2021. find our card
  12. Kandpal, N., Wallace, E., and Raffel, C. Deduplicating training data mitigates privacy risks in language models. In ICML, 2022. find our card
  13. Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. In EMNLP, 2020. find our card
  14. De Cao, N., Izacard, G., Riedel, S., and Petroni, F. Autoregressive entity retrieval. In ICLR, 2021. find
  15. Kassner, N., Krojer, B., and Schütze, H. Are pretrained language models symbolic reasoners over knowledge? In CoNLL, 2020. find
  16. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016. find our card
  17. Kwiatkowski, T., Palomaki, J., Rhinehart, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., et al. Natural Questions: A benchmark for question answering research. In TACL, 2019. find our card
  18. Rao, R. M., Liu, J., Verkuil, R., Meier, J., Canny, J., Abbeel, P., Sercu, T., and Rives, A. Msa transformer. In ICML, 2021. find
  19. Razeghi, Y., Logan IV, R. L., Gardner, M., and Singh, S. Impact of pretraining term frequencies on few-shot reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022, 2022. find
  20. Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Werra, L. V., Mou, C., Ponferrada, E. G., Nguyen, H., Frohberg, J., Šaško, M., Lhoest, Q., McMillan-Major, A., et al. The BigScience ROOTS corpus: A 1.6TB composite multilingual dataset. In NeurIPS, 2022. find
  21. Roberts, A., Raffel, C., and Shazeer, N. How much knowledge can you pack into the parameters of a language model? In EMNLP, 2020. find
  22. Lee, K., Chang, M.-W., and Toutanova, K. Latent retrieval for weakly supervised open domain question answering. In ACL, 2019. find our card
  23. Robertson, S. and Zaragoza, H. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in IR, 2009. find our card
  24. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating training data makes language models better. In ACL, 2021. find
  25. Scao, T. L., Fan, A., Akiki, C., Pavlick, E.-J., Ili’c, S., Hesslow, D., Castagn’e, R., Luccioni, A. S., Yvon, F., Gallé, M., Tow, J., Rush, A. M., et al. BLOOM: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. find
  26. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS, 2020. find
  27. Shin, S., Lee, S.-W., Ahn, H., Kim, S., Kim, H., Kim, B., Cho, K., Lee, G., Park, W., Ha, J.-W., et al. On the effect of pretraining corpora on in-context learning by a large-scale language model. In NAACL, 2022. find
  28. Liu, L., Lewis, P., Riedel, S., and Stenetorp, P. Challenges in generalization in open domain question answering. In Findings of NAACL, 2022. find
  29. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NeurIPS, 2017. find
  30. Mallen, A., Asai, A., Zhong, V., Das, R., Hajishirzi, H., and Khashabi, D. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511, 2022. find
  31. Wang, B. and Komatsuzaki, A. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https:// github.com/kingoflolz/mesh-transformer-jax, 2021. find
  32. Wang, T., Roberts, A., Hesslow, D., Scao, T. L., Chung, H. W., Beltagy, I., Launay, J., and Raffel, C. What language model architecture and pretraining objective work best for zero-shot generalization? In ICML, 2022. find
  33. Mendes, P. N., Jakob, M., Garcı́a-Silva, A., and Bizer, C. DBpedia Spotlight: Shedding light on the web of documents. In International Conference on Semantic Systems, 2011. find
  34. Wei, J., Garrette, D., Linzen, T., and Pavlick, E. Frequency effects on syntactic rule learning in transformers. In EMNLP, 2021. find
  35. Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S. Language models as knowledge bases? In EMNLP, 2019. find our card
  36. Petroni, F., Lewis, P. S. H., Piktus, A., Rocktäschel, T., Wu, Y., Miller, A. H., and Riedel, S. How context affects language models’ factual predictions. In AKBC, 2020. find our card
  37. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P. J., et al. Exploring the limits of transfer learning with a unified text-to-text transformer. In JMLR, 2020. find