Elamur Research⌕
☰
home / papers / 2016 / Understanding the Lexical Simplification Needs of Non-Native Speakers of English

Understanding the Lexical Simplification Needs of Non-Native Speakers of English

Paper · 2016 · Gustavo Paetzold and Lucia Specia

Summary

The paper asks which words actually cause trouble for non-native readers of English and what a good simpler replacement looks like for them. The authors ran three crowdsourced user studies, one for each stage of a typical Lexical Simplification pipeline: Complex Word Identification (CWI), Substitution Selection (SS) and Substitution Ranking (SR). Altogether, 1,100 non-native volunteers produced 211,564 annotations. The CWI sentences came from the CW Corpus, the LexMTurk Corpus and Simple Wikipedia. Annotations were then compared against 15 common complexity features: morphological, WordNet-based and n-gram language-model probabilities.

The main findings are:

  • Word length and syllable count say little about difficulty for this audience, unlike earlier results for dyslexic readers.
  • Words that readers find hard tend to have fewer senses and lower corpus probability.
  • n-gram probabilities that include the surrounding context predict simplicity better than plain word frequency, which contradicts an assumption common in older work.
  • Proficiency predicts how many words a reader struggles with, while native language predicts which ones.

The released datasets support building CWI models, simplifiers and readability tools tuned to non-native readers.

Key points

  • CWI study:
    • 9,200 sentences were annotated by 400 volunteers with 45 native languages, giving 158,624 annotations.
    • 200 sentences were each labelled by 20 people.
    • Even Simple Wikipedia sentences contained a word judged complex in 27.3% of cases.
  • CWI agreement: Kappa was 0.616 overall, far above earlier lexical substitution and simplification tasks (0.277 and 0.398). It rose to 0.718 among annotators sharing a native language.
  • SS study:
    • 2,554 sentences received 31,940 annotations.
    • Word-vector distance between target and candidate tracked grammaticality and meaning preservation better than language-model probability.
    • The probability of a candidate given the target's grammatical role was a reliable grammaticality signal.
    • Single-word replacements tended to damage multi-word expressions.
  • SR study:
    • 300 annotators made 21,000 pairwise judgements, with five per instance.
    • These were converted into 901 ranked instances using the Wauthier et al. (2013) algorithm.
    • The best feature was Simple[1,1], with Spearman −0.501 and TRank 0.593. Length reached only 0.172 and syllables 0.097.
  • Release: All data is available as a single download (User_Studies_NNS.zip).

Interesting ideas

  • I1. Annotate a small overlapping subset with many judges and the large remainder with one judge each, so agreement can be measured cheaply without multiplying labelling cost.
  • I2. Break agreement down by annotator subgroup (native language, age, proficiency); this revealed that disagreement is often systematic rather than noise.
  • I3. Collect easy binary comparisons ("which is simpler?") and infer full rankings from them algorithmically, instead of asking people to rank whole lists.
  • I4. Test widely held assumptions against the data, as the authors did when they showed that length is not a good difficulty cue for this audience and that context does matter.
  • I5. Check whether a replacement word is plausible in the grammatical role the original word fills, rather than only checking its overall frequency.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data could be labelled this way, with a small slice checked by all witness parsers and LLM teachers to calibrate trust in the single-witness bulk.
  • I2 (learn): Witness-parser agreement could be reported per genre or source, so that a "disagreement" caused by a dialect or domain is not mistaken for random error.
  • I3 (learn): LLM teacher tiers could judge pairs of candidate parses or answers, with a global ranking inferred afterwards, which is more reliable than asking for absolute scores.
  • I4 (learn): Heuristic features in repair rules could be audited for statistical significance before being trusted, similar to a negative control.
  • I5 (back): When generating or compressing text, a candidate lemma could be accepted only if it is attested in the same UD relation, and multi-word expressions should be protected from word-by-word substitution.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SheffieldSheffield · United Kingdom76 works · 20,808 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Developmental and Educational Psychology · 260 works

🧭 Topics 1 English · 1,373 works

Cite

Cite as Gustavo Paetzold and Lucia Specia. 2016. Understanding the Lexical Simplification Needs of Non-Native Speakers of English. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{paetzold2016understanding,
  title = {{Understanding the Lexical Simplification Needs of Non-Native Speakers of English}},
  author = {Gustavo Paetzold and Lucia Specia},
  booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
  year = {2016},
  doi = {10.5281/zenodo.1040781},
  url = {https://doi.org/10.5281/zenodo.1040781}
}

References

33 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandra Aluisio and Caroline Gasperin. 2010. Fostering digital inclusion and accessibility: The porsimples project for simplification of portuguese texts. In Proceedings of the 2010 NAACL Young Investigators Workshop on Computational Approaches to Languages of the Americas, pages 46–53. find
  2. Or Biran, Samuel Brody, and Noémie Elhadad. 2011. Putting it simply: a context-aware approach to lexical simplification. In Proceedings of the 49th ACL, pages 496–501. ↗ our card
  3. Marc Brysbaert and Boris New. 2009. Moving beyond kučera and francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for american english. Behavior research methods, 41:977–990. ↗
  4. Philip R Burns. 2013. Morphadorner v2: A java library for the morphological adornment of english language texts. Northwestern University, Evanston, IL. find
  5. Jean Carletta. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22:249–254. ↗ our card
  6. John Carroll, Guido Minnen, Darren Pearce, Yvonne Canning, Siobhan Devlin, and John Tait. 1999. Simplifying text for language-impaired readers. In Proceedings of the 9th EACL, pages 269–270. our card
  7. Raman Chandrasekar, Christine Doran, and Bangalore Srinivas. 1996. Motivations and methods for text simplification. In Proceedings of the 16th Conference on Computational Linguistics. our card
  8. Jan De Belder and Marie-Francine Moens. 2012. A dataset for the evaluation of lexical simplification. In Computational Linguistics and Intelligent Text Processing, pages 426–437. Springer. find
  9. Siobhan Devlin and John Tait. 1998. The use of a psycholinguistic database in the simplification of text for aphasic readers. Linguistic Databases, pages 161–173. find
  10. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Bradford Books. find
  11. Goran Glavaš and Sanja Štajner. 2015. Simplifying lexical simplification: Do we need simplified corpora? In Proceedings of the 53rd ACL, pages 63–69. our card
  12. Colby Horn, Cathryn Manduca, and David Kauchak. 2014. Learning a lexical simplifier using wikipedia. In Proceedings of the 52nd ACL, pages 458–463. our card
  13. Tomoyuki Kajiwara, Hiroshi Matsumoto, and Kazuhide Yamamoto. 2013. Selecting proper lexical paraphrase for children. In Proceedings of the 25th Rocling, pages 59–73. find
  14. David Kauchak. 2013. Improving text simplification language modeling using unsimplified text data. In Proceedings of the 51st ACL, pages 1537–1546. our card
  15. Dan Klein and Christopher D Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st ACL, pages 423–430. our card
  16. Diana McCarthy and Roberto Navigli. 2007. Semeval-2007 task 10: English lexical substitution task. In Proceedings of the 4th SemEval, pages 48–53. our card
  17. Rada Mihalcea, Ravi Sinha, and Diana McCarthy. 2010. Semeval-2010 task 2: Cross-lingual lexical substitution. In Proceedings of the 5th international workshop on semantic evaluation, pages 9–14. find
  18. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗ our card
  19. Bernardo Pereira Nunes, Ricardo Kawase, Patrick Siehndel, Marco a. Casanova, and Stefan Dietze. 2013. As simple as it gets - a sentence simplifier for different learning levels and contexts. In Proceedings of the 13th ICALT, pages 128–132. find
  20. Gustavo H. Paetzold and Lucia Specia. 2013. Text simplification as tree transduction. In Proceedings of the 9th STIL, pages 116–125. find
  21. Gustavo Henrique Paetzold and Lucia Specia. 2015. Lexenstein: A framework for lexical simplification. In Proceedings of The 53rd ACL, pages 85–90. find
  22. Gustavo Henrique Paetzold and Lucia Specia. 2016. Unsupervised lexical simplification for non-native speakers. In Proceedings of The 30th AAAI. find
  23. Gustavo Henrique Paetzold. 2015. Reliable lexical simplification for non-native speakers. In Proceedings of the 2015 NAACL Student Research Workshop, pages 9–16. find
  24. Luz Rello, Ricardo Baeza-Yates, Stefan Bott, and Horacio Saggion. 2013a. Simplify or help?: text simplification strategies for people with dyslexia. In Proceedings of the 10th W4A, pages 1–10. find
  25. Luz Rello, Ricardo Baeza-Yates, Laura Dempere-Marco, and Horacio Saggion. 2013b. Frequent words improve readability and short words improve understandability for people with dyslexia. Human-Computer Interaction, pages 203–219. find
  26. Luz Rello, Susana Bautista, Ricardo Baeza-Yates, Pablo Gervás, Raquel Hervás, and Horacio Saggion. 2013c. One half or 50%? an eye-tracking study of number representation readability. Human-Computer Interaction. find
  27. Thea Reves and Peter Medgyes. 1994. The non-native english speaking efl/esl teacher’s self-image: An international survey. System, 22(3):353–367. ↗
  28. Matthew Shardlow. 2013a. A comparison of techniques to automatically identify complex words. In Proceedings of the 51st ACL Student Research Workshop, pages 103–109. our card
  29. Matthew Shardlow. 2013b. The cw corpus: A new resource for evaluating the identification of complex words. In Proceedings of the 2nd Workshop on Predicting and Improving Text Readability for Target Reader Populations, pages 69–77. find
  30. Matthew Shardlow. 2014. Out in the open: Finding and categorising errors in the lexical simplification pipeline. In Proceedings of the 9th LREC, pages 1583–1590. find
  31. Lucia Specia, Sujay Kumar Jauhar, and Rada Mihalcea. 2012. Semeval-2012 task 1: English lexical simplification. In Proceedings of the 1st SemEval, pages 347–355. our card
  32. Andreas Stolcke. 2002. SRILM-an extensible language modeling toolkit. In Proceedings of the 2002 ICSLP, pages 257–286. find
  33. Fabian Wauthier, Michael Jordan, and Nebojsa Jojic. 2013. Efficient ranking from pairwise comparisons. In Proceedings of the 30th International Conference on Machine Learning, pages 109–117. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/paetzold-2016-understanding-lexical-simplification-needs-non/