Elamur Research⌕
☰
home / papers / 2012 / SemEval-2012 Task 1: English Lexical Simplification

SemEval-2012 Task 1: English Lexical Simplification

Summary

This paper presents the first shared task on English Lexical Simplification, organized at SemEval-2012. A full simplification pipeline has three stages: spotting difficult words, finding substitutes, and ordering those substitutes by simplicity. The task isolates the third stage, which the organizers see as the core of the problem. Systems receive a sentence, a target word and a set of substitutes already known to fit the context. They must rank the substitutes from simplest to most complex, and ties are allowed.

The data reuses the contexts and substitutes of the SemEval-2007 Lexical Substitution corpus. Fluent non-native English speakers re-annotated these with simplicity rankings. Systems were scored with a pairwise-comparison adaptation of kappa.

Nine systems took part. The strongest combined context-dependent and context-independent signals. However, the single most useful cue was how frequent a substitute is in general. A plain frequency baseline was very hard to outperform.

The task gives the field a common benchmark and evaluation protocol. It also shows that perceived simplicity is subjective and depends partly on context.

Key points

  • Data size and source: The data comes from the SemEval-2007 LS corpus, whose sentences were drawn from the Internet Corpus of English. It has 300 Trial and 1,710 Test contexts covering 201 mostly polysemous target words, with 10 contexts per word.
  • Annotators: Four annotators ranked the Trial set and five ranked the Test set. All were highly proficient second-language speakers with a university degree, and each annotated the entire dataset.
  • Inter-annotator agreement: Pairwise kappa was 0.386 on Trial and 0.398 on Test. Removing one outlier annotator raised the Test score to 0.443. The best-agreeing pair (0.52) shared the same language and educational background.
  • Evidence for context-dependency: On Test, 242 context pairs had an identical substitute list. In 139 of these, the human rankings differed. Overall, rankings changed in roughly 40–57% of such pairs.
  • Results: The top official kappa was 0.496. The "Simple Freq." baseline ranked near the top. On the post hoc top-rank metric, the best system (UOW-SHEF-SimpLex) reached 0.602, while the frequency baseline reached 0.585.

Interesting ideas

  • I1. Rank, don't label. Annotators order candidates relative to one another instead of assigning fixed difficulty categories. This avoids having to choose a number of classes and makes judgments comparative. The approach could suit any graded property, such as formality or readability of Ukrainian words.
  • I2. Gold standard by averaging ranks. Each candidate's rank is averaged across annotators, and candidates with equal averages are merged into one tied rank. This is a simple, transparent way to combine noisy subjective rankings into a single reference.
  • I3. Testing context-dependency with matching substitute lists. Contexts with identical substitute lists are treated as using the same word sense. Checking how often their rankings differ then gives a cheap, quantitative test of whether context matters, with no sense annotation needed.
  • I4. Complementary practical metrics. Pairwise kappa punishes small ordering differences, so the organizers also reported top-rank overlap and recall-at-n. They explicitly warn that these extra metrics reward outputs with many ties. Pairing a strict metric with a usage-oriented one, and checking for degenerate solutions, is a good general evaluation habit.
  • I5. Annotator background as a source of variance. Agreement was higher among annotators with shared first-language families or education. This suggests choosing annotators who match the intended audience profile, which matters when building resources for a specific reader group.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

97 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 91 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{specia2012semeval,
  title = {{SemEval-2012 Task 1: English Lexical Simplification}},
  author = {Specia, Lucia and Jauhar, Sujay Kumar and Mihalcea, Rada},
  booktitle = {*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)},
  year = {2012},
  url = {https://aclanthology.org/S12-1046/}
}

References

17 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marilisa Amoia and Massimo Romanelli. 2012. SB-mmSystem: Using Decompositional Semantics for Lexical Simplification. In English Lexical Simplification. Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), Montreal, Canada. find
  2. Or Biran, Samuel Brody, and Noemie Elhadad. 2011. Putting it simply: a context-aware approach to lexical simplification. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 496–501, Portland, Oregon. ↗ our card
  3. Thorsten Brants and Alex Franz. 2006. The google web 1t 5-gram corpus version 1.1. find
  4. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 22–64, Edinburgh, Scotland. find our card
  5. Arnaldo Candido, Jr., Erick Maziero, Caroline Gasperin, Thiago A. S. Pardo, Lucia Specia, and Sandra M. Aluisio. 2009. Supporting the adaptation of texts for poor literacy readers: a text simplification editor for Brazilian Portuguese. In Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications, pages 34–42, Boulder, Colorado. find
  6. J Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37–46, April. ↗
  7. Jan de Belder, Koen Deschacht, and Marie-Francine Moens. 2010. Lexical simplification. In Proceedings of Itec2010: 1st International Conference on Interdisciplinary Research on Technology, Education and Communication, Kortrijk, Belgium. find
  8. Siobhan Devlin and John Tait. 1998. The use of a psycholinguistic database in the simplification of text for aphasic readers. Linguistic Databases, pages 161– 173. find
  9. Sujay Kumar Jauhar and Lucia Specia. 2012. UOW-SHEF: SimpLex - Lexical Simplicity Ranking based on Contextual and Psycholinguistic Features. In English Lexical Simplification. Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), Montreal, Canada. find
  10. Anne-Laure Ligozat, Cyril Grouin, Anne Garcia-Fernandez, and Delphine Bernhard. 2012. ANNLOR: A Naive Notation-system for Lexical Outputs Ranking. In English Lexical Simplification. Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), Montreal, Canada. find
  11. Diana McCarthy and Roberto Navigli. 2007. Semeval-2007 task 10: English lexical substitution task. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), Prague, Czech Republic, pages 48–53. find
  12. E. Noreen. 1989. Computer-intensive methods for testing hypotheses. New York: Wiley. find
  13. Serge Sharoff. 2006. Open-source corpora: Using the net to fish for linguistic data. International Journal of Corpus Linguistics, 11(4):435–462. find
  14. Advaith Siddharthan. 2006. Syntactic simplification and text cohesion. Research on Language and Computation, 4:77–109. ↗
  15. Ravi Sinha. 2012. UNT-SimpRank: Systems for Lexical Simplification Ranking. In English Lexical Simplification. Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), Montreal, Canada. find
  16. Lucia Specia. 2010. Translating from complex to simplified sentences. In Proceedings of the 9th international conference on Computational Processing of the Portuguese Language, PROPOR’10, pages 30–39, Berlin, Heidelberg. Springer-Verlag. find
  17. Mark Yatskar, Bo Pang, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2010. For the sake of simplicity: unsupervised extraction of lexical simplifications from Wikipedia. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 365–368, Los Angeles, California. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/specia-2012-semeval-2012-task-1-english/