📄 Learning a Lexical Simplifier Using Wikipedia

Summary

The paper addresses lexical simplification: replacing difficult words with easier ones that keep the meaning and still fit the sentence. Earlier rule-based systems suffered from low coverage and from applying substitutions in the wrong context. The authors take 137K sentence pairs from English Wikipedia aligned with Simple English Wikipedia, word-align them with GIZA++, and filter the aligned word pairs to collect candidate substitutions. They then expand these rules with morphological variants. To choose a substitution in context, they train an SVM rank model with frequency, language-model and n-gram context features. The original word is included as one of the candidates, so the model can also decide to leave the word unchanged. For training and evaluation, they gathered 500 examples, each labelled by 50 Mechanical Turk workers. Their system reaches 76.1% precision and changes 86.3% of the examples, outperforming both a frequency baseline and the same ranker applied to the rules of Biran et al. The work shows that aligned parallel data can produce a much larger rule set without losing quality.

Key points

• Rule extraction:

  • Rules come from the Coster and Kauchak (2011) corpus of 137K normal/simple Wikipedia sentence pairs, word-aligned with GIZA++.
  • Three filters remove noise: stoplist words, pairs whose POS tags differ (Berkeley parser), and proper nouns.
  • MorphAdorner adds inflected forms of each rule.

• Rule set size: The extracted rules cover 14,478 words, with an average of 2.25 candidates per word. The Biran et al. rules cover 3,598 words, with 1.18 candidates per word.

• Ranking features: The SVM rank model uses:

  • the probability of a candidate given the word, estimated from the alignments;
  • unigram frequency in Simple English Wikipedia and in Google n-grams;
  • four language models built with SRILM using Kneser-Ney smoothing (Simple Wikipedia, English Wikipedia, an even interpolation of those two, and a Google 5-gram model);
  • Google n-gram counts of the candidate in a context window of one or two words on each side.

• Evaluation data: 500 examples, each annotated by 50 MTurk workers (25,000 annotations in total). Workers had to be US-based and have at least a 95% acceptance rate. The dataset is publicly released. Results are from 10-fold cross-validation.

• Results (precision / accuracy / changed):

  System	Precision	Accuracy	Changed
  rank-simplify	76.1%	66.3%	86.3%
  frequency baseline	53.9%	46.1%	84.9%
  rank-Biran	71.4%	3.4%	5.2%

  Precision@5 for rank-simplify is about 67%.

Interesting ideas

• I1. "No change" as a candidate: The original word is added to the candidate list, so deciding whether to simplify at all becomes part of the ranking itself. This removes the need for a separate classifier and could carry over to any substitution task, such as paraphrasing, normalization or terminology replacement.
• I2. Rules from aligned parallel text: Mining substitution rules from word alignments inside a monolingual sentence-aligned corpus produced a far larger rule set than mining unaligned comparable text. This is worth trying wherever simplified and original versions of documents exist side by side.
• I3. Cheap noise filters plus morphology: A few simple filters (POS agreement, stoplist, no proper nouns) clean the alignment noise. Symmetric morphological expansion, applied to both sides of a rule, then increases coverage. This is especially relevant for morphologically rich languages, where a rule learned from one word form would otherwise rarely match new text.
• I4. Crowd-ranked evaluation data: Collecting many free-text suggestions per item and ranking them by how often they occur yields graded gold data. That data supports metrics like precision@k as well as a separate "changed" rate that exposes systems with low coverage.
• I5. Separating rules from ranking: The ranker works independently of where the rules come from, as shown by applying it to the Biran et al. rules. This allows comparing or combining rule sources from different origins under the same ranking model.