📄 SENSEVAL-2: Overview

Summary

This overview reports on SENSEVAL-2, the second volunteer-run evaluation campaign for word sense disambiguation (WSD), held with ACL-2001 in Toulouse. Its goal was to compare WSD systems across many languages under shared conditions, without crowning a single winner. Three task types were offered: all-words tagging of running text, lexical sample tagging of preselected target words, and a Japanese translation-based task. Twelve languages were planned. Every task used a common XML encoding, a gold standard built by at least two annotators, and one central website at the University of Pennsylvania that enforced strict download and submission time limits. The campaign drew 93 systems from 34 teams. They were scored with precision, recall and coverage at fine, coarse and mixed sense granularity. A scorer bug found at the workshop led to all systems being rescored and to a controlled resubmission round. The paper matters because it documents the design of a large multilingual WSD benchmark, and it frankly lists what went wrong so later campaigns can avoid those problems.

Key points

• Planned languages were Czech, Dutch, English and Estonian (all-words); Basque, English, Italian, Japanese, Korean, Spanish and Swedish (lexical sample); Japanese (translation task); plus Chinese and Danish. Chinese, Danish and Dutch received no submissions because their data was not ready or was lost.
• English drew the most entries: 26 systems from 15 teams in the lexical sample task and 21 systems from 12 teams in all-words. Spanish had 12 submissions, and the Japanese translation task had 9.
• English tasks used WordNet 1.7 for the first time in SENSEVAL. Spanish, Italian and Estonian used versions of EuroWordNet.
• The design guidelines set several targets:
  • at least 5000 tagged words for all-words tasks;
  • at least 75+15n instances per lexical sample word, where n is the number of senses;
  • roughly 90% inter-tagger agreement;
  • a 2:1 split between training and test data.
• The competition ran from April 17 to June 18, 2001. Each task stayed open for at least 21 days. Teams had to submit within 7 days of downloading the test data and within 21 days of downloading the training data. Corrected resubmissions were accepted until 31 July 2001.

Interesting ideas

• I1. Scale data size with ambiguity. The 75+15n rule ties the number of annotated examples to how many senses a word has. This is an easy budgeting rule for any annotation project where label sets vary in size.
• I2. Use a larger test share than usual. A 2:1 train/test split exposes systems to more varied contexts. It is also fairer to methods that need no training data, which helps when supervised and unsupervised approaches compete.
• I3. Enforce time limits per participant. Logging timestamps for each team's downloads and capping the gap between test download and submission discourages tuning to the test items. It also lets teams join several tasks over a long campaign.
• I4. Score at several granularities. Accepting answers at group or hierarchy level, with partial credit for parent senses, separates real errors from harmless near-misses. The paper also warns that hierarchies and groupings must be consistent across tasks, or the coarse scores become meaningless.
• I5. Handle mistakes openly. When the scorer bug appeared, all systems were rescored, and resubmissions were clearly marked and required a written justification. Collecting system metadata at registration and leaving time for analysis before the workshop are also sensible lessons for organizing any shared task.