📄 SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoning

Summary

Evaluating how well LLMs actually understand long documents is difficult. Synthetic retrieval tests such as needle-in-a-haystack are easy for current models, and benchmarks built from existing corpora risk contamination and allow shortcut answers. SCALAR addresses this with academic papers whose citations act as expert-made labels, so no manual annotation is needed. The authors gather arXiv versions of accepted ICLR 2024 and ICLR 2025 papers, parse them into structured form, and mask in-text citations.

Models must recover the masked citations in two tasks:

• Multiple-choice QA (MCQA): pick one correct reference among four candidates.
• Cloze-style prediction: fill four masks at once from a shared candidate list.

Difficulty (Easy, Medium, Hard) is controlled by citation type, distractor sampling and context length. A live update mechanism adds new papers over time to limit contamination.

Human experts score above 90% on MCQA, while most LLMs struggle. The cloze task is harder still, with no model reaching 50%. SCALAR thus offers a realistic, renewable test of long-context reasoning in the scientific domain.

Key points

• A contamination analysis over four years of ICLR papers found that pre-2023 papers are often memorized by models, while 2024–2025 papers appear largely unseen. This finding motivated the benchmark's continuous updating.
• Difficulty is governed by several factors:
  • whether the citation is attributional or descriptive;
  • how distractor candidates are drawn (at random or by nearest neighbor);
  • question scope (section vs. full paper);
  • candidate representation (abstract, body, or full text).
• Two PhD candidates in computer science answered 30 MCQA questions spread across the three difficulty levels. They were perfect on Easy and highly accurate on Medium and Hard.
• In cloze prediction, open-source models lose about 10 points relative to MCQA, and proprietary models can lose up to half their score. The best average accuracy is below 40%.
• Results for Qwen2.5-7B-Instruct-1M:
  • On Easy, accuracy falls from 0.88 to 0.40–0.42 when candidates are given as full papers.
  • In the hardest setting (descriptive, nearest-neighbor, full text), it scores 22%, below the 25% chance level.
• Error analysis of 72 incorrect predictions: in over 70% of them, the model chose a topically similar but wrong paper.