📄 UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification

Summary

The paper describes UKP-Athene, the team's entry to the FEVER shared task, which ranked third of 23 systems in preliminary results. FEVER asks a system to check a claim against English Wikipedia. It must find relevant articles, pick out the evidence sentences, and then label the claim Supported, Refuted or NotEnoughInfo. A prediction only counts if the evidence is also correct. The authors built a three-stage pipeline:

• Document retrieval is treated as entity linking. Candidate mentions come from a constituency parser plus hand-written heuristics, articles are looked up through the MediaWiki API, and stemmed-title rules filter the results.
• Sentence selection uses ESIM reworked into a ranker trained with a hinge loss on sampled negatives.
• Entailment uses a second ESIM extension that reads five evidence sentences at once and combines them through attention.

On the development set, every stage beat the organizers' baseline by a wide margin, and the full-pipeline FEVER score roughly doubled. The work shows that a modular design with simple retrieval and attention-based aggregation of evidence goes a long way for automated fact-checking.

Key points

• The FEVER dataset has 185,445 Wikipedia-based claims, and each claim comes with one or more evidence sets.

• Development-set results against the baseline:

  Stage	Baseline	UKP-Athene
  Document retrieval accuracy	70.20	93.55
  Sentence selection recall	44.22	87.10
  Label accuracy	52.09	68.49
  FEVER score	32.27	64.74

• Keeping more MediaWiki hits per mention helped steadily. Moving from 3 to 7 results per mention raised document accuracy from 92.60 to 93.55.

• The entailment model scored best when given all five selected sentences (FEVER score 64.74). Sentence ranking at test time averaged the scores of ten ESIM models trained with different random seeds.

• Word vectors were GloVe and FastText concatenated, both pretrained on Wikipedia.

• The error analysis identified several recurring failure types:

  • misspelled entity names;
  • Wikipedia titles disambiguated with parentheses;
  • evidence that never names the claim's entity directly;
  • numbers the embeddings cannot tell apart, such as 22 versus 72 beats per minute;
  • borderline NotEnoughInfo cases.

Interesting ideas

• I1. Use more than named entities as mention candidates. Off-the-shelf NER misses titles of films, songs and similar works. The authors also treat every noun phrase, the words before the main verb, and the whole claim as candidates, which is a cheap way to boost recall for any task that links text to an encyclopedia.
• I2. Filter candidates by stemmed-title containment. An article is kept only if its title, with the parenthetical part removed and the words stemmed, appears fully inside the stemmed claim. This simple, explainable precision check could carry over to other languages, given a suitable stemmer.
• I3. Turn a classifier into a ranker. ESIM's final representation is fed to a single scoring neuron and trained with a margin loss. Positive examples are true evidence sets, and negatives are random sentences drawn from the same articles, so the negatives are hard but cheap to produce. The same trick works for any retrieval setting that only has positive annotations.
• I4. Aggregate multiple pieces of evidence with attention. Each claim–sentence pair is encoded separately, weighted by how similar it is to the claim, and the results are pooled. This lets the model judge a claim from several sentences without concatenating them into one long input, and it applies to multi-document QA or stance detection.
• I5. Sweep the number of retrieved items at each stage. Varying how many documents and sentences are passed downstream shows exactly where recall stops paying off. This is a useful diagnostic for tuning any multi-stage pipeline.