Elamur Research⌕
☰
home / papers / 2013 / A Comparison of Techniques to Automatically Identify Complex Words.

A Comparison of Techniques to Automatically Identify Complex Words.

Paper · 2013 · Matthew Shardlow

Summary

The paper tackles complex word (CW) identification, the first step of lexical simplification. Errors here carry forward: flagging too many words causes needless substitutions that can shift meaning, and flagging too few leaves hard words in the text. Shardlow noted that no direct comparison of CW identification techniques existed. He built an annotated corpus from Simple Wikipedia edit histories, where sentence pairs differing by a single simplified word expose the complex word, and added an equal number of negative (simple) examples. Three approaches were tested on this corpus: a "simplify everything" baseline, a frequency threshold learned from SUBTLEX counts, and an SVM using several lexical features. Frequency thresholding did not differ significantly from the naive baseline. The SVM gained a little precision but lost a great deal of recall. Precision was low for every method. The work matters because it offers a shared evaluation setup and shows that current CW identification still has much room for improvement.

Key points

  • Choosing a frequency source. Word frequency sources were compared on the SemEval 2012 Task 1 lexical simplification data:
    • SUBTLEX scored 0.3352.
    • The Wikipedia baseline scored 0.3270.
    • Kucera-Francis scored 0.3097.
    • A random baseline scored 0.0157.
    • For context, inter-annotator agreement on that data was 0.488.
  • SUBTLEX resource. SUBTLEX was chosen as the familiarity measure. It contains over 70,000 words drawn from subtitles of more than 8,000 American English films.
  • "Simplify everything" baseline. It follows Devlin and Tait (1998). It takes WordNet synonyms and substitutes the most frequent one by SUBTLEX count whenever that synonym is more frequent than the original word.
  • SVM setup.
    • It was trained with LIBSVM in Matlab, using an RBF kernel, a grid search over C and γ, and 5-fold cross validation.
    • Its features were SUBTLEX frequency, CD count, word length, syllable count, number of senses and number of synonyms.
  • Feature correlations. Frequency and CD count correlated moderately and positively with a word being simple. Length and syllable count correlated negatively. The SVM's results varied widely, which the author attributes to insufficient training data.

Interesting ideas

  • I1. Mining edit histories as annotation. Revision histories where an editor changed exactly one word give naturally occurring, in-context labels without commissioning new annotation. The same trick could work for any language with a simplified or edited wiki.
  • I2. Negatives from the same sentence. Negative examples are sampled at random from the very sentence containing the CW, on the logic that the editor left the other words untouched. Taking one negative per positive keeps classes balanced, so a classifier cannot win by labelling everything complex.
  • I3. Validating the frequency source first. Several frequency corpora are checked on an external benchmark before one is picked to drive the main experiments. This guards against building everything on a weak signal.
  • I4. Learning the threshold from data. The threshold is found by trying every cut point between adjacent words in the frequency-sorted training data and averaging the best one across folds. This is a simple, reproducible way to tune any single-score classifier.
  • I5. Correlation as feature analysis. Each feature is correlated with the binary label to see its direction and strength. This cheap analysis can guide feature design for readability or difficulty prediction in other languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ManchesterManchester · United Kingdom30 works · 43,942 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Developmental and Educational Psychology · 260 works

Cite

Cite as Matthew Shardlow. 2013. A Comparison of Techniques to Automatically Identify Complex Words. 51st Annual Meeting of the Association for Computational Linguistics Proceedings of the Student Research Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{shardlow2013comparison,
  title = {{A Comparison of Techniques to Automatically Identify Complex Words.}},
  author = {Matthew Shardlow},
  booktitle = {51st Annual Meeting of the Association for Computational Linguistics Proceedings of the Student Research Workshop},
  year = {2013},
  url = {https://aclanthology.org/P13-3015/}
}

References

20 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandra Maria Aluı́sio and Caroline Gasperin. 2010. Fostering digital inclusion and accessibility: the PorSimples project for simplification of Portuguese texts. In Proceedings of the NAACL HLT 2010 Young Investigators Workshop on Computational Approaches to Languages of the Americas, YIW-CALA ’10, p… find
  2. Stefan Bott, Luz Rello, Biljana Drndarevix, and Horacio Saggion. 2012. Can spanish be simpler? lexsis: Lexical simplification for spanish. In Coling 2012: The 24th International Conference on Computational Linguistics. find
  3. Thorsten Brants and Alex Franz. 2006. Web 1T 5-gram Version 1. find
  4. Marc Brysbaert and Boris New. 2009. Moving beyond Kucera and Francis : a critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behav Res Methods. find
  5. John Carrol, Guido Minnen, Yvonne Canning, Siobhan Devlin, and John Tait. 1998. Practical simplification of english newspaper text to assist aphasic readers. In AAAI-98 Workshop on Integrating Artificial Intelligence and Assistive Technology. find
  6. Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27. Software available at http://www.csie.ntu.edu.tw/˜cjlin/libsvm. ↗
  7. Jan De Belder and Marie-Francine Moens. 2012. A dataset for the evaluation of lexical simplification. In Computational Linguistics and Intelligent Text Processing, volume 7182 of Lecture Notes in Computer Science, pages 426–437. Springer Berlin / Heidelberg. find
  8. Siobhan Devlin and John Tait. 1998. The use of a psycholinguistic database in the simplification of text for aphasic readers. Linguistic Databases, pages 161–173. find
  9. Noémie Elhadad. 2006. Comprehending technical texts: Predicting and defining unfamiliar terms. In AMIA Annual Symposium proceedings, volume 2006, page 239. American Medical Informatics Association. find
  10. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Bradford Books. find
  11. Caroline Gasperin, Lucia Specia, Tiago Pereira, and Sandra M. Aluı́sio. 2009. Learning when to simplify sentences for natural text simplification. In Encontro Nacional de Inteligência Artificial. find
  12. Julia Hancke, Sowmya Vajjala, and Detmar Meurers. 2012. Readability classification for German using lexical, syntactic, and morphological features. In Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012), pages 1063–1080, Mumbai, India. find
  13. Sujay Kumar Jauhar and Lucia Specia. 2012. Uow-shef: Simplex – lexical simplicity ranking based on contextual and psycholinguistic features. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume… find
  14. Partha Lal and Stefan Rüger. 2002. Extract-based summarization with simplification. In Proceedings of the ACL. find
  15. Keith Rayner and Susan Duffy. 1986. Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & Cognition, 14:191–201. find
  16. Lucia Specia, Sujay Kumar Jauhar, and Rada Mihalcea. 2012. Semeval-2012 task 1: English lexical simplification. In First Joint Conference on Lexical and Computational Semantics. our card
  17. S. Rebecca Thomas and Sven Anderson. 2012. Wordnet-based lexical simplification of a document. In Jeremy Jancsary, editor, Proceedings of KON-VENS 2012, pages 80–88. ÖGAI, September. Main track: oral presentations. find
  18. Mark Yatskar, Bo Pang, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2010. For the sake of simplicity: Unsupervised extraction of lexical simplifications from Wikipedia. In Proceedings of the NAACL. our card
  19. Qing Zeng, Eunjung Kim, Jon Crowell, and Tony Tse. 2005. A text corpora-based estimation of the familiarity of health terminology. Biological and Medical Data Analysis, pages 184–192. find
  20. Qing Zeng-Treitler, Sergey Goryachev, Tony Tse, Alla Keselman, and Aziz Boxwala. 2008. Estimating consumer familiarity with health terminology: a context-based approach. Journal of the American Medical Informatics Association, 15(3):349–356. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shardlow-2013-comparison-techniques-automatically-identify-complex/