📄 SemEval-2010 Task 5 : Automatic Keyphrase Extraction from Scientific Articles

Summary

This paper describes SemEval-2010 Task 5, a shared task on automatically assigning keyphrases to scientific articles. Earlier keyphrase work used many different datasets and splits, so results were hard to compare. The organizers built a benchmark of 284 papers from conferences and workshops, all taken from the ACM Digital Library. The papers were drawn evenly from four ACM subject classes and converted from PDF to plain text. Each paper has two kinds of gold keyphrases: those the authors supplied and those added by student annotators ("readers").

Participating systems returned a ranked list of 15 keyphrases per document. Organizers scored the lists by exact match against Porter-stemmed gold phrases, using micro-averaged precision, recall and F-score at cutoffs of 5, 10 and 15. Nineteen systems took part, and most of them beat the TF×IDF and classifier baselines. The best system, HUMB, scored F = 27.5% on the combined gold set.

The authors also estimated how high scores could realistically go. They found substantial room for improvement and argued that both the fixed-size outputs and the strict matching metric understate real performance. The dataset became a public reference point for the field.

Key points

• Data:
  • 40 trial, 144 training and 100 test articles; the trial set is a subset of training.
  • Articles come from ACM classes C2.4, H3.3, I2.11 and J4, spread evenly across splits.
  • The test set has 387 author-assigned, 1,217 reader-assigned and 1,482 combined keyphrases.
• Reader annotation: 50 NUS Computer Science students each labelled 5 papers. In the test data, 15% of reader keyphrases and 19% of author keyphrases never appear in the text. This caps achievable recall at 85% and 81% respectively.
• Baselines: TF×IDF, Naïve Bayes and Maximum Entropy over 1–3-gram candidates. At top 15, the TF×IDF baseline reached F = 15.1% on combined keyphrases and 12.9% on reader keyphrases.
• Participation: 73 teams downloaded the trial data and 21 entered; two withdrew, leaving 19 ranked systems. HUMB led with F = 23.5% on reader keyphrases, 27.5% on combined and 19.3% on author keyphrases.
• Human ceiling: Only 125 of the 387 author keyphrases match reader keyphrases exactly. Reader annotations measured against author ones give an F-score of about 33.6%, a rough human upper bound.

Interesting ideas

• I1. Score against labels that cannot leak. Author keyphrases can be copied straight from the original PDFs, so the organizers focused evaluation on reader keyphrases collected separately and held out. Any benchmark whose labels might be scraped from the source should have a gold set like this.
• I2. Compute the theoretical ceiling before reading the scores. The organizers measured how many gold phrases actually occur in the text, which gives the maximum recall an extractive system can reach. They also used agreement between readers and authors as a human ceiling. Together these make modest absolute scores much easier to interpret.
• I3. Pin down the normalization tools. Participants were told in advance exactly which Porter stemmer implementation would be applied. This removes score differences caused only by preprocessing, which is cheap insurance in any shared task using string matching.
• I4. Allow controlled surface variants with human review. Genitive rewrites such as "A of B" → "B A" were accepted as matches. Borderline cases where the meaning changes were decided by a committee rather than by a blanket rule. This is a middle ground between strict exact match and fully semantic evaluation.
• I5. Learn from the design limitations. The authors suggest that future tasks let systems output a variable number of keyphrases per document. They also recommend evaluating with semantic similarity rather than exact match. Both are useful reminders when designing extraction metrics for other tasks or languages.