📄 A Benchmark Dataset for Automatic Detection of Claims and Evidence in the Context of Controversial Topics

Summary

Argumentation mining has lacked high-quality annotated data. This paper introduces a benchmark corpus of claims and evidence tied to specific controversial topics, which earlier corpora such as AraucariaDB and the V/IP Corpus did not do. Rather than the full Toulmin model, the authors use a deliberately minimal scheme. A Context Dependent Claim (CDC) is a concise, general statement that backs or opposes a debate motion. A Context Dependent Evidence (CDE) is a passage that supports a given CDC with respect to that motion. Each CDE is tagged as Study, Expert, or Anecdotal. About 20 trained in-house annotators worked in the GATE environment on English Wikipedia articles. The pipeline had separate stages for search, detection, and confirmation, each with five people working independently, and an item was kept only if at least three confirmed it. The final dataset contains 2,683 argument elements across 33 motions. Pairwise kappa was moderate, and estimated recall was about 90%. Because the resource covers many topics, it can support topic-independent tools for finding arguments.

Key points

• The 33 motions were sampled at random from the idebate.org database. Annotators labeled 586 Wikipedia articles from the April 3, 2012 dump and found 1,392 CDCs in 321 of them.
• Evidence annotation was completed for only 12 motions, covering 350 CDCs in 104 articles. This produced 1,291 CDEs: 431 Study, 516 Expert, and 529 Anecdotal. A CDE can carry several types; for example, 118 are both Study and Expert.
• A candidate counts as a CDC only if it meets all five criteria:
  • Strength
  • Generality
  • Phrasing
  • Keeping text spirit
  • Topic unity
• CDEs use the same criteria minus generality. Annotators could make one minor permitted fix, such as resolving an anaphor, and the corrected version is included in the data.
• Average pairwise Cohen's kappa was 0.39 for claim confirmation and 0.4 for evidence confirmation. Estimated average recall was 90.6% for CDCs and 90.0% for CDEs.

Interesting ideas

• I1. Detect, then confirm. One group of annotators casts a wide net for candidates, and a separate group votes on the pooled list. This trades some noise at the start for a majority filter at the end, which suits fuzzy annotation targets like stance or argumentativeness.
• I2. Rejections must name a criterion. An annotator who rejects a candidate has to cite which rule it breaks. This makes disagreements easier to diagnose and gives concrete material for refining the guidelines.
• I3. Recall without a gold standard. The authors assume each annotator independently finds some fixed fraction of the true items in an article. From the overlap between annotators' detections, they infer how many items exist in total and therefore how many were missed. The same approach could estimate coverage in any crowd- or team-based extraction task.
• I4. Agreement when annotators don't overlap fully. Since no two annotators had identical assignments, kappa was averaged over annotator pairs, and pairs with fewer than 100 shared items were dropped. This is a practical fallback when standard multi-rater measures cannot be applied.
• I5. Consensus answer keys for ongoing feedback. Experienced annotators first solved selected tasks independently, then agreed on a reference answer. Everyone else was compared against it and received a feedback report roughly every week. This keeps quality monitoring feasible even when full review of every task is too expensive.