Elamur Research⌕
☰
home / papers / 2022 / Controllable Lexical Simplification for English

Controllable Lexical Simplification for English

Paper · 2022 · Kim Cheng Sheang, Daniel Ferrés and Horacio Saggion

Summary

The paper tackles lexical simplification (LS), the task of replacing a hard word in context with an easier one. Earlier neural LS systems, notably LSBert, generate candidates with a masked language model. This work asks whether fine-tuning a sequence-to-sequence Transformer with explicit control signals can do as well. The authors build ConLS by fine-tuning T5 on the small TSAR-EN dataset. They add three control tokens to the front of each source sentence:

  • a character-length ratio between substitute and complex word,
  • a frequency-rank ratio based on FastText vocabulary order,
  • a coarse encoding of how annotators ranked the candidate.

Special markers enclose the target word in both input and output. At inference, the length and rank values are tuned once on validation data and then frozen. On LexMTurk, BenchLS and NNSeval, ConLS performs on par with a modified LSBert that returns ten candidates, and beats it on several metrics. An ablation shows that using all tokens together works best. This is presented as the first controllable approach to LS.

Key points

  • Training and validation use TSAR-EN, which has 386 instances, each with 25 crowd-sourced substitutes, split 90/10. Testing uses LexMTurk (500 Wikipedia sentences), BenchLS and NNSeval.
  • The control tokens are WL (word length ratio), WR (inverse-frequency rank ratio from FastText) and CR (candidate rank, mapped to 1.00/0.75/0.50/0.25/0.00).
  • Every gold candidate becomes its own parallel training pair.
  • Model size matters a great deal. On LexMTurk, ACC@1 is 23.40 for T5-Small, 60.00 for T5-Base and 80.60 for T5-Large.
  • In the ablation on LexMTurk, all tokens together reach ACC@1 80.60 and ACC@1@Top1 43.80. Without tokens the scores are 79.20 and 41.80.
  • WL and WR are searched with Optuna in the range 0.5–1.25 in steps of 0.05, with CR fixed at 1.00.
  • The LSBert baseline was adapted to output 10 candidates (15 mask selections), and candidates identical to the complex word were removed.

Interesting ideas

  • I1. Control values are computed from the data as ratios between source and target properties, such as length and frequency. They are attached as prefix tokens, so at generation time one can request outputs that are "shorter" or "more common" without any post-filtering.
  • I2. Bracket markers around the target span, in both input and output, let the system reliably extract the substituted word from free-form generated text.
  • I3. One annotated item with many ranked answers is expanded into many training pairs, each carrying its rank as a coarse token. The model thus learns the ordering itself rather than relying on a separate ranker.
  • I4. Control values are not predicted per instance. Instead, they are tuned once on held-out data to maximize a chosen metric and then frozen. This offers a cheap baseline before building a dedicated predictor.
  • I5. The ablation over token subsets shows that most single tokens add little and that the gain from all tokens together is only one to two points on a 500-item test. This is a reminder to report small-margin effects carefully.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): When generating text from the meaning graph, lexical choice could be steered by explicit, logged ratio knobs for frequency and length. This would make each simplification decision traceable to a stated target value.
  • I2 (learn): When LLM teacher tiers rewrite or relabel sentences, wrapping the focus span in markers would let us align the teacher's answer to a specific tree node instead of guessing by string matching.
  • I3 (learn): Silver data agreed by several witness parsers could be expanded the same way. Each alternative analysis would be tagged with its agreement level, so the learner sees both the consensus and the weaker variants.
  • I4 (learn): Thresholds in self-induced repair rules or confidence cutoffs could first be set by a validation sweep and frozen. A per-sentence predictor would only be adopted if it beats this baseline.
  • I5 (learn): Evaluations of new features should include a "no feature" negative control and subset ablations, so that small gains are not over-interpreted as real improvements.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universitat Pompeu FabraBarcelona · Spain36 works · 11,433 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 English · 1,373 works

Cite

Cite as Kim Cheng Sheang, Daniel Ferrés and Horacio Saggion. 2022. Controllable Lexical Simplification for English. Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.tsar-1.19

@inproceedings{sheang2022controllable,
  title = {{Controllable Lexical Simplification for English}},
  author = {Kim Cheng Sheang, Daniel Ferrés and Horacio Saggion},
  booktitle = {Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.tsar-1.19},
  url = {https://doi.org/10.18653/v1/2022.tsar-1.19}
}

References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, T. Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. our card
  2. Colby Horn, Cathryn Manduca, and David Kauchak. 2014. Learning a Lexical Simplifier Using Wikipedia. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 458–463, Baltimore, Maryland. Association for Computational Linguistics. our card
  3. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. CoRR, abs/1910.13461. ↗ our card
  4. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  5. Louis Martin, Éric de la Clergerie, Benoît Sagot, and Antoine Bordes. 2020. Controllable sentence simplification. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4689– 4698. ↗
  6. Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoît Sagot. 2022. MUSS: Multilingual unsupervised sentence simplification by mining paraphrases. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1651–1664, Marseille, France. European Language Resources Association. our card
  7. Gustavo Paetzold and Lucia Specia. 2016a. Benchmarking Lexical Simplification Systems. In Proceedings of LREC-2016. find
  8. Gustavo Paetzold and Lucia Specia. 2016b. Unsupervised Lexical Simplification for Non-Native Speakers. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). find
  9. Gustavo Paetzold and Lucia Specia. 2017a. A Survey on Lexical Simplification. Journal of Artificial Intelligence Research, 60:549–593. ↗
  10. Gustavo Paetzold and Lucia Specia. 2017b. Lexical simplification with neural ranking. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 34–40, Valencia, Spain. Association for Computational Linguistics. find
  11. Jipeng Qiang, Yun Li, Zhu Yi, Yunhao Yuan, and Xindong Wu. 2020. Lexical simplification with pre-trained encoders. Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 8649—-8656. find
  12. Jipeng Qiang, Yun Li, Yi Zhu, Yunhao Yuan, Yang Shi, and Xindong Wu. 2021. Lsbert: Lexical simplification based on bert. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3064– 3076. ↗
  13. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗ our card
  14. Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. our card
  15. Mark Yatskar, Bo Pang, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2010. For the sake of simplicity: Unsupervised extraction of lexical simplifications from Wikipedia. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 365–368, Los Angeles, California. Association for Computational Linguistics. ↗ our card
  16. Seid Muhie Yimam, Chris Biemann, Shervin Malmasi, Gustavo H. Paetzold, Lucia Specia, Sanja Stajner, Anaïs Tack, and Marcos Zampieri. 2018. A Report on the Complex Word Identification Shared Task 2018. CoRR, abs/1804.09132. ↗
  17. Sanja Štajner, Richard Evans, Constantin Orasan, and Ruslan Mitkov. 2012. What can readability measures really tell us about text complexity. In Proceedings of workshop on natural language processing for improving textual accessibility, pages 14–22. Citeseer. find
  18. Sanja Štajner, Daniel Ferrés, Matthew Shardlow, Kai North, Marcos Zampieri, and Horacio Saggion. 2022a. Lexical Simplification Benchmarks for English, Portuguese, and Spanish. Frontiers in Artificial Intelligence, 5. ↗
  19. Sanja Štajner, Kim Cheng Sheang, and Horacio Saggion. 2022b. Sentence simplification capabilities of transfer-based models. Proceedings of the AAAI Conference on Artificial Intelligence, 36(11):12172– 12180. find
  20. Horacio Saggion. 2017. Automatic Text Simplification. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. ↗
  21. Matthew Shardlow. 2014. A Survey of Automated Text Simplification. International Journal of Advanced Computer Science and Applications, 4. ↗
  22. Kim Cheng Sheang and Horacio Saggion. 2021. Controllable sentence simplification with a unified text-to-text transfer transformer. In Proceedings of the 14th International Conference on Natural Language Generation, pages 341–352, Aberdeen, Scotland, UK. Association for Computational Linguistics. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sheang-2022-controllable-lexical-simplification-english/