📄 The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task

Summary

The FEVEROUS shared task, held at the FEVER 2021 workshop, extends automated fact checking from purely textual evidence to a mix of text and structured Wikipedia content such as tables and lists. Given a human-written claim, a system has to find supporting material (sentences and table cells, with their page, section and header context) and then decide whether the claim is SUPPORTED, REFUTED or NOT ENOUGH INFO. Unlike FEVER 2018, NEI predictions also require partial evidence, and claims are considerably longer. Submissions were ranked by the FEVEROUS score, which counts a claim as correct only when the label is right and a full gold evidence set appears among the returned items. The organisers supplied a baseline built from TF-IDF and entity matching, a sequence-labelling cell extractor and a RoBERTa NLI verdict model. Of the systems submitted, six outperformed this baseline, and the winner, "Bust a move!", scored 27%. NEI claims and table-based reasoning remained the weakest areas, so the benchmark leaves plenty of room for improvement. The scoring platform stays open for follow-up work.

Key points

• FEVEROUS has 87,026 claims, split into 71,291 train, 7,890 dev and 7,845 blind test. Claims average 25.3 words, compared with 9.4 in FEVER, and evidence may be sentences only, cells only, or both.
• Systems may return at most 5 sentences and 25 table cells (including captions and list items) per claim. Anything beyond these limits is dropped without penalty.
• The task received 13 entries, hosted on EvalAI. "Bust a move!" won with a score of 0.27, about 9 points above the baseline's 0.18. Papelo came second with 0.26 and achieved the best result on tabular-only claims.
• NEI made up only about 5% of the training data. The two top systems produced no NEI predictions at all, and their per-class F1 for NEI was 0.00.
• Evidence recall matters far more here than in FEVER: the FEVEROUS score correlates with recall at ρ = 0.97, against 0.53 for FEVER. Claim-only bias is also much weaker, with a claim-only model reaching 58% against a 56% majority baseline (in FEVER, 62% vs 33%).