Elamur Research
☰
home / papers / 2007 / SemEval-2007 Task 07: Coarse-Grained English All-Words Task

SemEval-2007 Task 07: Coarse-Grained English All-Words Task

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/S07-1006.pdf

Summary

Fine-grained sense distinctions in WordNet are widely seen as a major barrier to accurate Word Sense Disambiguation (WSD). Even human annotators often disagree on them, and systems at Senseval-3 reached only about 65% accuracy. For SemEval-2007, the organizers built an all-words evaluation that uses a coarser grouping of WordNet senses. They clustered WordNet 2.1 senses by linking them to top-level entries of the Oxford Dictionary of English (ODE). For most words this linking was automatic, using the SSI algorithm. For every word in the test data, an expert lexicographer did the linking by hand. The resulting test set covers five texts from different domains. Coarse clustering roughly halved average polysemy, and annotator agreement rose well above typical fine-grained levels. Fourteen systems from 12 teams took part. The top scores were around 82–83% F1, only a few points above a strong most-frequent-sense (MFS) baseline. The task shows that sense granularity strongly shapes both human and system performance. It also leaves open whether coarse senses actually help downstream applications.

Key points

  • Test data: 5,377 running words from five articles were used, and 2,269 open-class word instances were annotated.
    • Three articles came from WSJ.
    • One was the Wikipedia entry on computer programming.
    • One was an excerpt from Knights of the Art.
  • Polysemy: average polysemy fell from 6.18 (WordNet) to 3.06 (coarse inventory), and about 30% of instances became monosemous.
  • Inter-annotator agreement:
    • 86.44% on the sense mapping (590 senses).
    • 93.80% on test-set annotation (710 instances).
    • Fine-grained annotation typically reaches only about 67–72%.
  • Baselines and results:
    • The MFS baseline (SemCor-based) scored 78.89% and the random baseline 52.43%.
    • The best ranked system, NUS-PT, reached 82.50% F1.
    • The organizer-submitted UoR-SSI, listed separately from the ranking, reached 83.21%.
    • Every system that beat MFS used it as a backoff.
  • Domain effects: MFS and most supervised systems did worse on the computer science and biography texts. Their predominant senses differ from those in SemCor.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics: Word sense disambiguation

Cited by

74 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 71 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{navigli2007semeval,
  title = {{SemEval-2007 Task 07: Coarse-Grained English All-Words Task}},
  author = {Navigli, Roberto and Litkowski, Kenneth C. and Hargraves, Orin},
  booktitle = {Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)},
  year = {2007},
  url = {https://aclanthology.org/S07-1006/}
}

References

9 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tim Chklovski and Rada Mihalcea. 2002. Building a sense tagged corpus with open mind word expert. In Proc. of ACL O THER UC TR MFS CS √ √ √ × √ × √ √ √ Parallel corpus × √ × × √ × × √ × × √ × × √ × × × × × × × × SSI LKB × × × × × √ √ × √ √ √ √ √ √ × √ × × × × × √ × × × × × × × × × × × √ × × × √ × √ √ √ × × × × × × × √ √ √ × × × √ √ √ √ √ × × × × × × √ × × √ × × × × √ × × × × 2002 Workshop on WSD: Recent Successes and Future Directions. Philadelphia, PA. find
  2. Philip Edmonds and Adam Kilgarriff. 2002. Introduction to the special issue on evaluating word sense disambiguation systems. Journal of Natural Language Engineering, 8(4):279– 291. find
  3. Christiane Fellbaum, editor. 1998. WordNet: an Electronic Lexical Database. MIT Press. find
  4. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Comp. Volume, pages 57– 60, New York City, USA. find
  5. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Proceedings of the ARPA Workshop on Human Language Technology, pages 303–308, Princeton, NJ, USA. find
  6. Roberto Navigli and Paola Velardi. 2005. Structural semantic interconnections: a knowledge-based approach to word sense disambiguation. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 27(7):1063–1074. find
  7. Roberto Navigli. 2006. Meaningful clustering of senses helps boost word sense disambiguation performance. In Proc. of the 44th Annual Meeting of the Association for Computational Linguistics joint with the 21st International Conference on Computational Linguistics (COLING-ACL 2006), pages 105–112. Sydney, Australia. find
  8. Benjamin Snyder and Martha Palmer. 2004. The english all-words task. In Proc. of ACL 2004 SENSEVAL-3 Workshop, pages 41–43. Barcelona, Spain. find
  9. Catherine Soanes and Angus Stevenson, editors. 2003. Oxford Dictionary of English. Oxford University Press. find