📄 WildGraphBench: Benchmarking GraphRAG with Wild-Source Corpora

Summary

Most GraphRAG benchmarks use short, pre-cleaned passages. That makes it hard to judge how graph-based retrieval copes with long, messy, mixed-source document collections, the setting where it should help most. WildGraphBench fills this gap using Wikipedia. Articles from 12 top-level topics are sampled. The external pages each article cites (mostly news, plus magazines, research sites, PDFs and books) form the retrieval corpus. The article's citation-backed sentences serve as gold facts.

From these, the authors build about 1,100 questions of three kinds:

• single-fact lookups,
• multi-fact questions that combine several references,
• section-level summaries.

Answers are scored against the gold statements. QA uses accuracy, and summaries use statement-level precision, recall and F1.

The authors test flat RAG baselines and several GraphRAG systems. Graph methods add little on single-fact queries but clearly help when facts must be combined across a moderate number of sources. All methods score poorly on summaries, partly because graph aggregation favours high-level claims over specific details. The benchmark exposes realistic weaknesses that curated datasets hide.

Key points

• Corpus and gold data: Wikipedia reference pages make up the corpus, and about 78% of them are news sources. Citation-linked Wikipedia statements serve as ground truth. The 12 topics are Wikipedia's top-level content categories, with "Reference" excluded.

• Systems evaluated: NaiveRAG, BM25, Fast-GraphRAG, HippoRAG2, MS GraphRAG (local/global), LightRAG and LinearRAG. Indexing costs vary widely: on the Culture domain, MS GraphRAG takes about 21.5 h to index, compared with seconds for the flat baselines.

• Model swap test: Replacing gpt-4o-mini with gpt-5-mini in HippoRAG2 (Culture domain) gives large QA gains:

  • single-fact: 62.79 → 80.23
  • multi-fact: 24.32 → 40.54
  • summary F1: only 6.68 → 8.33

  This points to retrieval as the bottleneck for summaries.

• Retrieval budget and human reference: For summaries, F1 rises with retrieval top-k up to k = 8 and then falls as noise increases. Human annotators reach 15.30 F1 on the People subset.

• Error analysis and judge reliability:

  • Multi-fact errors are 65% generation-side and 35% retrieval-side.
  • Half of summary errors come from gold evidence that was never retrieved.
  • The LLM judge agrees with human checks 89.8%, 96.0% and 98.0% of the time across the three question types.