📄 Crowdsourced Corpus with Entity Salience Annotations

Summary

Detecting which entities a document is really about (entity salience) needs training data. The existing resources, the Microsoft Document Aboutness (MDA) and New York Times (NYT) datasets, omit the article texts because of copyright. They also depend on proprietary NER output, and NYT derives its salience labels automatically. The authors address this by building what they describe as the first openly available corpus with human salience judgments. They started from the Reuters-128 news corpus (a subset of Reuters-21578), added common entities detected by Entityclassifier.eu, and used CrowdFlower workers to label each entity as most salient, less salient or not salient. The resource is released as RDF in NIF 2.0 under CC0. To demonstrate its use, they trained supervised classifiers on two kinds of features: local ones (position, mention counts, entity type) and global ones from DBpedia (PageRank, HITS, in/out-degree, property counts). The full Random Forest model reached F1 0.607, beating positional and frequency baselines. Performance degraded only moderately when entity links were deliberately corrupted. The corpus gives salience research an open, reusable benchmark.

Key points

• Corpus size: 128 Reuters economic articles with 4,429 entity mentions (878 named entities from Reuters-128 plus 3,551 common entities) and 2,024 unique entities, of which 3,194 mentions are linked to DBpedia.
• Label distribution: 804 most salient (18%), 1,750 less salient (40%) and 1,875 not salient (42%).
• Crowdsourcing:
  • Three judgments were collected per entity over 36 days, totalling 18,058 judgments.
  • Only contributors with a trust score above 70% were kept, leaving 14,528 trusted judgments.
  • Inter-annotator agreement was 63.66%.
• Classifier results: With ten-fold cross-validation, Random Forest using all local and global features achieved P 0.612, R 0.608 and F1 0.607.
• Robustness to linking errors: Corrupting 10% of entity links lowered F1 by 2.47%, and with 50% wrong links F1 still reached 0.553. Link-independent features largely compensate for the noise.