Elamur Research⌕
☰
home / papers / 2011 / Simple English Wikipedia: A New Text Simplification Task

Simple English Wikipedia: A New Text Simplification Task

Summary

Sentence simplification means rewriting a sentence with easier vocabulary and structure while keeping its meaning. Data-driven methods have been held back by a lack of training material. Earlier simplification and compression datasets held only a few thousand pairs and mostly covered deletion.

Coster and Kauchak build a much larger parallel resource from May 2010 dumps of English Wikipedia and Simple English Wikipedia. Their pipeline has three stages:

  • Articles: paired by title, after removing stubs, disambiguation pages, single-line pages and meta-pages.
  • Paragraphs: linked using TF-IDF cosine similarity.
  • Sentences: aligned within linked paragraphs by a dynamic program that allows skips and small many-to-many matches, then filtered by a similarity cutoff.

The result is 137K sentence pairs, orders of magnitude more than previous resources, and human checks found most sampled pairs correctly aligned. A word-alignment analysis shows that rewording, deletion, reordering, merging and splitting are all frequent.

Treating simplification as monolingual translation, the authors train Moses on the corpus. It slightly but significantly beats leaving sentences unchanged, and an n-best oracle points to considerable remaining headroom. The publicly released corpus makes statistical simplification research feasible.

Key points

  • Corpus construction:
    • Filtering left 10,588 article pairs, a 90% drop from about 110K Simple English Wikipedia articles.
    • These yielded 75K aligned paragraphs and 137K sentence pairs.
    • That is roughly 14 pairs per article, although a simple article averages about 40 sentences.
  • Alignment quality:
    • At a similarity threshold of 0.5, two judges reached consensus that 91 of 100 sampled pairs were correct.
    • Raising the threshold to 0.75 left 90K pairs, with 98 of 100 judged correct.
    • 27% of pairs are identical on both sides; these were deliberately kept.
  • Sentence-level alignment operations:
    • 27% skip a simple sentence and 23% skip a normal sentence.
    • 37% are 1-to-1, 8% are one normal to two simple, and 5% are two normal to one simple.
    • No 2-to-2 alignments were found, and 65% of simple paragraphs have no normal counterpart.
  • Word-level transformations (from GIZA++ alignments): rewordings occur in 65% of pairs, deletions in 47%, reorderings in 34%, merges in 31% and splits in 27%.
  • Moses experiment:
    • Setup: 124K training pairs, a language model trained on the simple side, a 500-pair development set and a 1300-pair test set.
    • BLEU scores were 0.5937 for no simplification (NONE), 0.5987 for Moses and 0.6317 for Moses-Oracle.
    • Word-F1 and SSA follow the same pattern.
    • All differences are significant at p = 0.01 (bootstrap resampling).
    • T3 was also tried but performed poorly.

Interesting ideas

  • I1. Coarse-to-fine alignment of comparable corpora. Matching documents first, then paragraphs, then sentences narrows the search space at each step. It needs no manual annotation, so it can mine parallel text from any pair of loosely related collections, such as standard and simplified versions of a text.
  • I2. Similarity filter instead of a built-in mismatch penalty. Discarding pairs below a cutoff after alignment is easy to understand and lets you regenerate the data at different precision/size trade-offs. This makes it simple to study how noise in training data affects a model.
  • I3. Keeping unchanged pairs and comparing against a do-nothing baseline. Leaving identical pairs in teaches a system when not to edit. Reporting the "no change" score shows how much real gain a model delivers, which matters for any task where the input is already close to the output.
  • I4. Oracle selection from an n-best list. Choosing the best of the top 1000 candidates against the reference gives an upper bound on what better reranking or tuning could achieve. It is a cheap diagnostic for any generation system.
  • I5. Profiling a corpus through word-alignment statistics. Counting rewordings, deletions, reorderings, splits and merges from automatic alignments reveals which operations a task actually involves. This can guide model choice before any system is built.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

144 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 126 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{coster2011simple,
  title = {{Simple English Wikipedia: A New Text Simplification Task}},
  author = {Coster, William and Kauchak, David},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  url = {https://aclanthology.org/P11-2117/}
}

References

30 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Regina Barzilay and Noemie Elhadad. 2003. Sentence alignment for monolingual comparable corpora. In Proceedings of EMNLP. find
  2. John Carroll, Gido Minnen, Yvonne Canning, Siobhan Devlin, and John Tait. 1998. Practical simplification of English newspaper text to assist aphasic readers. In Proceedings of AAAI Workshop on Integrating AI and Assistive Technology. find
  3. Raman Chandrasekar and Bangalore Srinivas. 1997. Automatic induction of rules for text simplification. In Knowledge Based Systems. ↗
  4. David Chiang. 2010. Learning to translate with source and target syntax. In Proceedings of ACL. find our card
  5. James Clarke and Mirella Lapata. 2006. Models for sentence compression: A comparison across domains, training requirements and evaluation measures. In Proceedings of ACL. find
  6. Trevor Cohn and Mirella Lapata. 2009. Sentence compression as tree transduction. Journal of Artificial Intelligence Research. ↗
  7. Lijun Feng. 2008. Text simplification: A survey. CUNY Technical Report. find
  8. Michel Galley and Kathleen McKeown. 2007. Lexicalized Markov grammars for sentence compression. In Proceedings of HLT/NAACL. ↗
  9. Ruifang Ge and Raymond Mooney. 2006. Discriminative reranking for semantic parsing. In Proceedings of COLING. find
  10. Siddhartha Jonnalagadda, Luis Tari, Jorg Hakenberg, Chitta Baral, and Graciela Gonzalez. 2009. Towards effective sentence simplification for automatic processing of biomedical text. In Proceedings of HLT/NAACL. find
  11. Dan Klein and Christopher Manning. 2003. Accurate unlexicalized parsing. In Proceedings of ACL. find our card
  12. Kevin Knight and Daniel Marcu. 2002. Summarization beyond sentence extraction: A probabilistic approach to sentence compression. Artificial Intelligence. ↗
  13. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of ACL. find
  14. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of EMNLP. find
  15. Ryan McDonald. 2006. Discriminative sentence compression with soft syntactic evidence. In Proceedings of EACL. find
  16. Makoto Miwa, Rune Saetre, Yusuke Miyao, and Jun’ichi Tsujii. 2010. Entity-focused sentence simplication for relation extraction. In Proceedings of COLING. find
  17. Courtney Napoles and Mark Dredze. 2010. Learning simple Wikipedia: A cogitation in ascertaining abecedarian language. In Proceedings of HLT/NAACL Workshop on Computation Linguistics and Writing. find
  18. Rani Nelken and Stuart Shieber. 2006. Towards robust context-sensitive sentence alignment for monolingual corpora. In Proceedings of AMTA. find
  19. Tadashi Nomoto. 2007. Discriminative sentence compression with conditional random fields. In Information Processing and Management. ↗
  20. Tadashi Nomoto. 2008. A generic sentence trimmer with CRFs. In Proceedings of HLT/NAACL. find
  21. Tadashi Nomoto. 2009. A comparison of model free versus model intensive approaches to sentence compression. In Proceedings of EMNLP. find
  22. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. ↗
  23. Franz Och and Hermann Ney. 2004. The alignment template approach to statistical machine translation. Computational Linguistics. ↗
  24. Franz Josef Och, Kenji Yamada, Stanford U, Alex Fraser, Daniel Gildea, and Viren Jain. 2004. A smorgasbord of features for statistical machine translation. In Proceedings of HLT/NAACL. find
  25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of ACL. find our card
  26. Emily Pitler. 2010. Methods for sentence compression. Technical Report MS-CIS-10-20, University of Pennsylvania. find
  27. Jenine Turner and Eugene Charniak. 2005. Supervised and unsupervised learning for sentence compression. In Proceedings of ACL. find
  28. David Vickrey and Daphne Koller. 2008. Sentence simplification for semantic role labeling. In Proceedings of ACL. find
  29. Elif Yamangil and Rani Nelken. 2008. Mining Wikipedia revision histories for improving sentence compression. In ACL. find
  30. Mark Yatskar, Bo Pang, Critian Danescu-Niculescu-Mizil, and Lillian Lee. 2010. For the sake of simplicity: Unsupervised extraction of lexical simplifications from Wikipedia. In HLT/NAACL Short Papers. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/coster-2011-simple-english-wikipedia-new-text/