Elamur Research⌕
☰
home / papers / 2024 / Towards Content Accessibility Through Lexical Simplification for Maltese as a Low-Resource Language

Towards Content Accessibility Through Lexical Simplification for Maltese as a Low-Resource Language

Summary

Maltese is a low-resource language, and the only earlier automatic simplifier for it, a 2014 n-gram/bag-of-words system, rarely produced usable substitutes. The authors built a lexical simplification (LS) system that replaces difficult words in Maltese news text with easier ones. Their four-stage pipeline is adapted from LSBert, which needs no supervised training data, only a masked language model, here BERTu. The stages are potential complex word identification, substitute generation, substitute selection, and substitute ranking. To evaluate and tune it, they built a small annotated set from four Maltese news portals. Annotators checked candidates against a thesaurus and a Word2Vec model, a professional proofreader reviewed them, and native speakers judged which ones were simpler. In a single-blind study with 207 people, meaning was preserved in roughly half of the simplified sentences. When meaning survived, most outputs were judged simpler than or as simple as the original. The work offers a reusable recipe for LS in languages with few resources. It also includes a browser extension, MaltEasy, as a reference implementation.

Key points

  • Data set: 80 news sentences, 4 per category per site, covering five categories (commerce, sports, lifestyle, politics, general) across four portals.
    • Sentence lengths were restricted to the typical 10–25 token range.
    • The set was split 40/40 into dev and test.
    • Target words were content words, excluding named entities and English words.
  • Simplicity annotation: 4 native annotators made binary judgements, and the scores were tallied and divided by 4. The original word competed as a candidate too.
    • The dev set had 280 targets, 228 of which kept the original among the simplest options.
    • The dev set was undersampled to 52 vs 52 for balance.
  • Ranking features: a LightGBM model ranks substitutes. By split importance, the strongest features were embedding similarity (3039) and PLL score (3031), then shuffled-corpus frequency (2057), character count, simplification-corpus frequency, and press-corpus frequency.
  • Substitute ranking (SR) results:
    • On system-generated candidates: accuracy 0.886, F-score 0.667, Prec.@1 0.724.
    • On gold candidates: accuracy 0.766, F-score 0.737, Prec.@1 0.814.
  • Human study:
    • Raters judged meaning identical in 44% of pairs.
    • Within those pairs, 53% preferred the generated version, 29% the original, and 18% were unsure.
    • For sentences the system left unchanged, 73% agreed they could not be simplified further.
    • Meaning changes usually came from a single bad substitution.

Interesting ideas

  • I1. When judging simplicity, the original word is a candidate in its own right, so "leave it alone" is a legitimate winning answer rather than a failure. — For Mova (back): when the generator or the text → graph → text compressor picks among paraphrases, it could always score the unchanged wording too. A rewrite would then happen only when some option clearly beats the original.
  • I2. Annotators found candidates using tools from a different model family (Word2Vec, a thesaurus) than the system itself (BERT). This keeps the gold standard from quietly favouring the system's own outputs. — For Mova (learn): gold or silver sets for evaluating gparse or the perceptron parser should be seeded by witnesses unrelated to the model under test. Otherwise agreement partly measures self-similarity.
  • I3. After a word is swapped, the attached Maltese article clitic is recomputed, because its form depends on the new word's first sound. Substitution is treated as a lexical change plus a surface repair step. — For Mova (back): when generation or translation swaps a lexeme, a final pass could fix dependent forms such as "a/an", agreement, or contractions. Validity would then not depend on the substitute happening to fit.
  • I4. In the human evaluation, raters first judged whether meaning was kept, and simplicity was analysed only within the meaning-preserving cases. A fluent but wrong output therefore cannot count as a success. — For Mova (back): the compressor's round-trip evaluation could first gate on graph equivalence. Only then would it compare brevity or readability of the regenerated text.
  • I5. The system's abstentions (sentences it chose not to simplify) were checked separately with humans, which treats declining to act as a decision to be evaluated. — For Mova (answer): the readers could log their "cannot answer / no change needed" outcomes as their own category. That category could be audited against human judgements alongside normal accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing · Society & ethics

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.ltedi-1.5

@inproceedings{meli2024content,
  title = {{Towards Content Accessibility Through Lexical Simplification for Maltese as a Low-Resource Language}},
  author = {Meli, Martina and Tanti, Marc and Porter, Chris},
  booktitle = {Proceedings of the Fourth Workshop on Language Technology for Equality, Diversity, Inclusion},
  year = {2024},
  doi = {10.18653/v1/2024.ltedi-1.5},
  url = {https://doi.org/10.18653/v1/2024.ltedi-1.5}
}

References

13 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eric Mutabazi and Nathanaël Wallenhorst. 2020. Une citoyenneté de seconde classe? n’ayons pas peur des mots! bildungsforschung, (1):1–13. find
  2. Rodrigo Alarcon, Lourdes Moreno, and Paloma Martínez. 2021. Lexical Simplification System to Improve Web Accessibility. IEEE Access, 9:58755– 58767. find
  3. Irina Nikishina, Alsu Vakhitova, Elena Tutubalina, and Alexander Panchenko. 2022. Cross-Modal Contextualized Hidden State Projection Method for Expanding of Taxonomic Graphs. In Proc. of TextGraphs-16: Graph-based Methods for Natural Lang. Process., pages 11–24. Association for Computational Linguistics. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. of the 2019 NAACL-HLT., Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. find
  5. Jenny Ortiz-Zambrano and Arturo Montejo-Ráez. 2021. SINAI at SemEval-2021 Task 1: Complex word identification using Word-level features. In Proc. of the 15th Int. Workshop on Semant. Eval., pages 126–129, Bangkok, Thailand. find
  6. Pierre Finnimore, Elisabeth Fritzsch, Daniel King, Alison Sneyd, Aneeq Ur Rehman, Fernando Alva-Manchego, and Andreas Vlachos. 2019. Strong Baselines for Complex Word Identification across Multiple Languages. In Proc. of the 2019 NAACL-HLT, pages 970–977, Minneapolis, Minnesota. Association for Computational Linguistics. find
  7. G. H. Paetzold and L. Specia. 2016. Unsupervised Lexical Simplification for non-native speakers. In AAAI’16: Proc. of the Thirtieth AAAI Conf. on AI, pages 3761–3768. find
  8. Sian Gooding and Ekaterina Kochmar. 2019. Recursive context-aware lexical simplification. In Proc. of the 2019 Conf. on Empirical Methods in Natural Lang. Proc. and the 9th Int. Joint Conf. on Natural Lang. Proc. (EMNLP-IJCNLP), pages 4853–4863, Gustavo H. Paetzold and Lucia Specia. 2017a. A Survey on Lexical Simplification. J. Artif. Intell. Res., 60:549–593. find
  9. Gustavo Henrique Paetzold and Lucia Specia. 2017b. Lexical Simplification with Neural Ranking. In Proc. of the 15th Conf. of the Eur. Chapter of the Assoc. for Comput. Linguistics: Volume 2, Short Papers, volume 2, pages 34–40. Parameter Values SG module Max. masks 1, 2, 3 Beam size 3, 4, 5 Pro-clitic mask * yes, no Mask fill order LTR, RTL Use CSR yes, no SS module POS tag filtering yes, no Similarity filtering yes, no Similarity method cosine, WMD Scaling yes, no SR module Candidate words source SS, SG Similarity method cosine, WMD Feature norm. yes, no ML model logistic reg., XGBoost, LightGBM, Naïve Bayes Jipeng Qiang, Yun Li, Yi Zhu, Yunhao Yuan, Yang Shi, and Xindong Wu. 2021. LSBert: Lexical Simplification Based on BERT. IEEE/ACM Trans. on Audio Speech and Lang. Process., 29:3064–3076. find
  10. Jipeng Qiang, Xinyu Lu, Yun Li, Yunhao Yuan, and Xindong Wu. 2020. Chinese Lexical Simplification. IEEE/ACM Transactions on Audio Speech and Language Processing, 29:1819–1828. find
  11. Eva Rolin, Quentin Langlois, Patrick Watrin, and Thomas François. 2021. FrenLyS: A Tool for the Automatic Simplification of French General Language Texts. In RANLP 2021, pages 1196–1205. INCOMA Ltd. find
  12. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proc. of the 58th Annu. Meeting of the Assoc. for Comput. Linguistics. Association for Computational Linguistics. *Table 4: Hyperparameter search space used when tuning the separate modules. * Only for the Maltese data set.* R. Sathyaraj and Prabu Sevugan. 2015. An approach for software fault prediction to measure the quality of different prediction methodologies using software metrics. Indian J. of Sci. and Tech,, 8. find our card
  13. Mario Serracino-Inglott. 2016. Id-Dizzjunarju Malti, 4 edition. Merlin Publishers. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/meli-2024-content-accessibility-through-lexical-simplification/