Elamur Research⌕
☰
home / papers / 2024 / Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries

Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries

Paper · 2024 · Kiran Vodrahalli; Santiago Ontanon; Nilesh Tripuraneni; Kelvin Xu; Sanil Jain; Rakesh Shivanna; Jeffrey Hui; Nishanth Dikkala; Mehran Kazemi; Bahare Fatemi; Rohan Anil; Ethan Dyer; Siamak Shakeri; Roopali Vij; Harsh Mehta; Vinay Ramasesh; Quoc Le; Ed Chi; Yifeng Lu; Orhan Firat; Angeliki Lazaridou; Jean-Baptiste Lespiau; Nithya Attaluri; and Kate Olszewska

Summary

Widely used long-context tests, such as Needle-in-a-Haystack, mainly check whether a model can locate one or a few facts. That reveals little about whether it can combine information scattered across a very large input. Existing reasoning benchmarks have their own problems: short contexts, costly manual scaling, data likely leaked into training, or shortcuts that let a model skip most of the context.

The authors propose the Latent Structure Queries (LSQ) framework. A long input conceals a structure among irrelevant material, and the model must strip away the noise and answer questions about that hidden structure. Using LSQ, they build Michelangelo, a compact synthetic benchmark that can be regenerated automatically and scored easily. It has three tasks set in code and natural language: Latent List, MRCR and IDK.

Testing ten frontier models up to 128K tokens, and the Gemini 1.5 family up to 1M, reveals a steep early drop, often before 32K, followed by a flatter curve. Long-context synthesis therefore remains largely unsolved. Michelangelo offers a leakage-resistant way to track progress on it.

Key points

  • Latent List: the model tracks a short Python list through operations (append, insert, pop, remove, sort, reverse) hidden among no-op filler. It then reports a slice, sum, min, max or length of the result. Complexity levels are 1, 5 or 20 relevant operations, using values drawn from [−4000, 4000].
  • MRCR and IDK: MRCR (multi-round coreference resolution) requires reproducing a specific earlier part of the context while respecting order. IDK tests whether the model can recognize that the context does not contain the answer.
  • Scale of evaluation: ten frontier models were evaluated up to 128K tokens, and Gemini 1.5 Pro/Flash up to 1M. GPT and Claude models stay non-trivial up to 128K, while Gemini models generalize as far as 1M.
  • Family resemblance: models from the same family (Gemini, GPT, Claude) show parallel MRCR curves, most strikingly Claude 3.5 Sonnet and Claude 3 Opus.
  • Recommendation: the authors propose MRCR as a default replacement for Needle-in-a-Haystack. It uses a simple metric, is robust to prompt choice, and gives a useful signal even for smaller and non-post-trained models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Kiran Vodrahalli; Santiago Ontanon; Nilesh Tripuraneni; Kelvin Xu; Sanil Jain; Rakesh Shivanna; Jeffrey Hui; Nishanth Dikkala; Mehran Kazemi; Bahare Fatemi; Rohan Anil; Ethan Dyer; Siamak Shakeri; Roopali Vij; Harsh Mehta; Vinay Ramasesh; Quoc Le; Ed Chi; Yifeng Lu; Orhan Firat; Angeliki Lazaridou; Jean-Baptiste Lespiau; Nithya Attaluri; and Kate Olszewska. 2024. Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries. arXiv:2409.12640.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.12640 · arXiv 2409.12640

@misc{attaluri2024michelangelo,
  title = {{Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries}},
  author = {Kiran Vodrahalli; Santiago Ontanon; Nilesh Tripuraneni; Kelvin Xu; Sanil Jain; Rakesh Shivanna; Jeffrey Hui; Nishanth Dikkala; Mehran Kazemi; Bahare Fatemi; Rohan Anil; Ethan Dyer; Siamak Shakeri; Roopali Vij; Harsh Mehta; Vinay Ramasesh; Quoc Le; Ed Chi; Yifeng Lu; Orhan Firat; Angeliki Lazaridou; Jean-Baptiste Lespiau; Nithya Attaluri; and Kate Olszewska},
  year = {2024},
  doi = {10.48550/arxiv.2409.12640},
  eprint = {2409.12640},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.12640}
}

References

30 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Agarwal, A. Singh, L. M. Zhang, B. Bohnet, L. Rosias, S. Chan, B. Zhang, A. Anand, Z. Abbas, A. Nova, J. D. Co-Reyes, E. Chu, F. Behbahani, A. Faust, and H. Larochelle. Many-shot in-context learning, 2024. URL https://arxiv.org/abs/2404.11018 . ↗
  2. M. Akter, N. Bansal, and S. K. Karmaker. Revisiting automatic evaluation of extractive summarization task: Can we do better than ROUGE? In S. Muresan, P. Nakov, and A. Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 1547–1560, Dublin, Ireland, May 2… find
  3. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, L. E. Shafey, Y. Huang, K. Meier-Hellstern, G. Mishra, E. Moreira, M. Omernick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G. H. Abrego, J. Ahn, J. Austin,… — PaLM 2 Technical Report
  4. Anthropic. Model Card and Evaluations for Claude Models, 2023. find
  5. Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku, 2024a. find
  6. Anthropic. Claude 3.5 Sonnet Model Card Addendum, 2024b. find
  7. S. Arora, S. Eyuboglu, A. Timalsina, I. Johnson, M. Poli, J. Zou, A. Rudra, and C. Ré. Zoology: Measuring and improving recall in efficient language models, 2023. URL https://arxiv.org/ abs/2312.04927 . ↗
  8. Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023.
  9. B. Bohnet, A. Nova, A. T. Parisi, K. Swersky, K. Goshvadi, H. Dai, D. Schuurmans, N. Fiedel, and H. Sedghi. Exploring and benchmarking the planning capabilities of large language models, 2024a. URL https://arxiv.org/abs/2406.13094 . — [arXiv:2406.13094](https://arxiv.org/abs/2406.13094) ↗
  10. B. Bohnet, K. Swersky, R. Liu, P. Awasthi, A. Nova, J. Snaider, H. Sedghi, A. T. Parisi, M. Collins, A. Lazaridou, O. Firat, and N. Fiedel. Long-span question-answering: Automatic question generation and qa-system ranking via side-by-side evaluation, 2024b. URL https://arxiv.org/abs/2406. 00179 . ↗
  11. Y. Chang, K. Lo, T. Goyal, and M. Iyyer. Booookscore: A systematic exploration of book-length summarization in the era of LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=7Ttk3RzDeu . ↗
  12. A. Cohan and N. Goharian. Revisiting summarization evaluation for scientific articles. In N. Calzolari, K. Choukri, T. Declerck, S. Goggi, M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis, editors, Proceedings of the Tenth International Conference on Language Res… find
  13. O. Goldman, A. Jacovi, A. Slobodkin, A. Maimon, I. Dagan, and R. Tsarfaty. Is it really long context if all you need is retrieval? towards genuinely difficult long context nlp, 2024. URL https://arxiv.org/abs/2407.00402 . ↗
  14. Google, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, S. Mariooryad, Y. Ding, X. Geng, F. Alcober, R. Frostig, M. Omernick, L. Walker, C. Paduraru, C. Sorokin, A. Tacchetti, C. Gaffney, S. Daruki, O. Sercinoglu, Z. Gleicher, J. Love, P. Voigtlaender,… — Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
  15. C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. Ruler: What’s the real context size of your long-context language models?, 2024. URL — RULER: What's the Real Context Size of Your Long-Context Language Models? https://arxiv.org/ abs/2404.06654 . ↗
  16. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack/ blob/main/README.md . M. Karpinska, K. Thai, K. Lo, T. Goyal, and M. Iyyer. One thousand and one pairs: A "novel" challenge for long-context language models, 2024. URL https://arxiv.org/abs/2406.16264 . ↗
  17. D. Kaushik and Z. C. Lipton. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010–5015,…
  18. Y. Kim, Y. Chang, M. Karpinska, A. Garimella, V. Manjunatha, K. Lo, T. Goyal, and M. Iyyer. Fables: Evaluating faithfulness and content selection in book-length summarization, 2024. URL https://arxiv.org/abs/2404.01261 . — [arXiv:2404.01261](https://arxiv.org/abs/2404.01261) ↗
  19. T. Kočiskỳ, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328, 2018.
  20. Y. Kuratov, A. Bulatov, P. Anokhin, I. Rodkin, D. Sorokin, A. Sorokin, and M. Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack, 2024. URL https://arxiv.org/ abs/2406.10149 . ↗
  21. J. Lee, A. Chen, Z. Dai, D. Dua, D. S. Sachan, M. Boratko, Y. Luan, S. M. R. Arnold, V. Perot, S. Dalmia, H. Hu, X. Lin, P. Pasupat, A. Amini, J. R. Cole, S. Riedel, I. Naim, M.-W. Chang, and K. Guu. Can long-context language models subsume retrieval, rag, sql, and more?, 2024. URL https://arxiv.org… ↗
  22. M. Levy, A. Jacoby, and Y. Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848, 2024.
  23. M. Li, S. Zhang, Y. Liu, and K. Chen. Needlebench: Can llms do retrieval and reasoning in 1 million context window?, 2024. URL https://arxiv.org/abs/2407.11963 . — [arXiv:2407.11963](https://arxiv.org/abs/2407.11963) ↗
  24. Magic. Hashhop: Long context evaluation. https://github.com/magicproduct/hash-hop , 2024. ↗
  25. M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035, 2023.
  26. OpenAI. GPT-4 Technical Report, Mar. 2023. URL https://arxiv.org/abs/2303.08774 . ↗
  27. OpenAI. GPT-4o System Card, 2024.
  28. Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006, 2020.
  29. Wikipedia. Blind men and an elephant — Wikipedia, the free encyclopedia. http://en.wikipedia.org/w/index.php?title=Blind%20men%20and%20an%20elephant&oldid=1242776766 , 2024. [Online; accessed 18-September-2024]. ↗
  30. X. Zhang, Y. Chen, S. Hu, Z. Xu, J. Chen, M. K. Hao, X. Han, Z. L. Thai, S. Wang, Z. Liu, et al. ∞-bench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13718, 2024. — [arXiv:2402.13718](https://arxiv.org/abs/2402.13718) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vodrahalli-2024-michelangelo-long-context-evaluations-haystacks/