Elamur Research⌕
☰
home / papers / 2019 / Multiattentive Recurrent Neural Network Architecture for Multilingual Readability Assessment

Multiattentive Recurrent Neural Network Architecture for Multilingual Readability Assessment

Paper · 2019 · Ion Madrazo Azpiazu and Maria Soledad Pera

Summary

Readability assessment tools usually depend on large sets of hand-crafted, language-specific features. This makes them hard to port to new languages and hard to compare, so practitioners still default to simple formulas like Flesch. The authors propose Vec2Read, a deep learning model that takes raw tokens as its main input and is enriched with POS and morphological tags.

A single word-level bidirectional LSTM encodes the text. A multiattentive mechanism then scores sentences and words from three linguistic angles (semantic, syntactic, morphological) and uses those scores to weight word representations into a length-normalized text vector. The model handles binary, multilevel, and continuous readability scales.

It was evaluated on data sets in seven languages and compared against traditional formulas, feature-based state-of-the-art systems, and other neural architectures. The authors report that it performs strongly enough to serve as a new, much tougher baseline than Flesch.

The work matters because it shows language-agnostic readability prediction is feasible. Its attention scores also offer an intuitive way to explain which words and sentences make a text hard.

Key points

  • The seven languages tested are English, Spanish, French, Italian, Dutch, Catalan, and Basque. The authors present it as the first readability approach evaluated on more than two languages; earlier cross-lingual work covered only Dutch and English.
  • Input is a triple of tokens, POS tags, and morphological tag sets. Each tag set includes every morphological feature for the language, with NA where a feature doesn't apply. Embeddings can be random or pretrained.
  • The architecture uses one word-level BiLSTM rather than a hierarchical word-plus-sentence RNN. Sentence structure enters only through the two-level attention, and per-data-set weights for the three attention types are learned (Table 4).
  • Attention scores can be decomposed by perspective and visualized, for example on a Wikipedia snippet about Qatna and a Spanish sentence. This positions attention as a more accessible alternative to feature dumps such as Coh-Metrix.
  • Code and evaluation framework are public at github.com/ionmadrazo/Vec2Read. Exact performance figures were not part of the provided excerpt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Boise State UniversityBoise · United States3 works · 93 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksMorphology · 405 works

Cite

Cite as Ion Madrazo Azpiazu and Maria Soledad Pera. 2019. Multiattentive Recurrent Neural Network Architecture for Multilingual Readability Assessment. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00278

@article{azpiazu2019multiattentive,
  title = {{Multiattentive Recurrent Neural Network Architecture for Multilingual Readability Assessment}},
  author = {Ion Madrazo Azpiazu and Maria Soledad Pera},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.1162/tacl\_a\_00278},
  url = {https://doi.org/10.1162/tacl_a_00278}
}

References

45 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abbasi, Hsinchun Chen, and Arab Salem. 2008. Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. Association of Computer Machinery Transactions on Information Systems (TOIS), 26(3):12. find
  2. Rebekah George Benjamin. 2012. Reconstructing readability: Recent developments and recommendations in the analysis of text difficulty. Educational Psychology Review, 24(1):63–88. find
  3. Elmer V. Bernstam, Dawn M. Shelton, Muhammad Walji, and Funda Meric-Bernstam. 2005. Instruments to assess the quality of health information on the World Wide Web: What can our patients actually use? International Journal Medical Information, 74(1):13–19. find
  4. Amani A Al-Ajlan, Hend S Al-Khalifa, and A Al-Salman. 2008. Towards the development of an automatic readability measurements for arabic language. In Proceedings of the International Conference on Digital Information Management, pages 506–511. find
  5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146.
  6. Bharat Ram Ambati, Siva Reddy, and Mark Steedman. 2016. Assessing relative sentence complexity using an incremental CCG parser. In Proceedings of the 2016 Conference of the Rudolph Flesch. 1948. A new readability yardstick. Journal of Applied Psychology, 32(3):221. find
  7. Yaw-Huei Chen, Yi-Han Tsai, and Yu-Ta Chen. 2011. Chinese readability assessment using tf-idf and svm. In Proceedings of the International Conference on Machine Learning and Computing, volume 2, pages 705–710. find
  8. Thomas François and Cédrick Fairon. 2012. An ai readability formula for french as a foreign language. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 466–477.
  9. Alice Davison and Robert N. Kantor. 1982. On the failure of readability formulas to define readable texts: A case study from adaptations. Reading Research Quarterly, 17:187–209. find
  10. Orphée De Clercq and Véronique Hoste. 2016. All mixed up? Finding the optimal feature set for general readability prediction and its application to english and dutch. Computational Linguistics, 42(3):457–490. find
  11. Goran Glavaš and Sanja Štajner. 2015. Simplifying lexical simplification: Do we need simplified corpora? In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), vo…
  12. Felice Dell’Orletta, Simonetta Montemagni, and Giulia Venturi. 2011. Read-it: Assessing readability of italian texts with a view to text simplification. In Workshop on Speech and Language Processing for Assistive Technologies, pages 73–83.
  13. Itziar Gonzalez-Dios, Marı́a Jesús Aranzabe, Arantza Dı́az de Ilarraza, and Haritz Salaberri. 2014. Simple or complex? Assessing the readability of Basque texts. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 334–344. find
  14. Joel Denning, Maria Soledad Pera, and Yiu-Kai Ng. 2016. A readability level prediction tool for k-12 books. Journal of the Association for Information Science and Technology, 67(3):550–565. find
  15. Arthur C. Graesser, Danielle S. McNamara, and Jonna M. Kulikowich. 2011. Coh-metrix: Providing multilevel analyses of text characteristics. Educational Researcher, 40(5):223–234. find
  16. W.H. Douma. 1960. De leesbaarheid van landblouwbladen een onderzoek naar en een toepassing van lees baarheidsformules. afd. Socio-logie en Sociographie van de Landbouwhogeschool te Wageningen, Bulletin 17. find
  17. Stephen Grossberg. 1988. Nonlinear neural networks: Principles, mechanisms, and architectures. Neural Networks, 1(1):17–61. find
  18. Andrew Elfenbein. 2011. Research in text and the uses of coh-metrix. Educational Researcher, 40(5):246–248. find
  19. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of Advances in Neural Information Processing Systems, pages 1693–1701.
  20. Lijun Feng, Martin Jansche, Matt Huenerfauth, and Noémie Elhadad. 2010. A comparison of features for automatic readability assessment. In Proceedings of the International Conference on Computational Linguistics: Posters, pages 276–284.
  21. Sepp Hochreiter. 1991. Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Universität München, 91:1. find
  22. José Fernández Huerta. 1959. Medidas sencillas de lecturabilidad. Consigna, 214:29–32. find
  23. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  24. P. R. Fitzsimmons, B. D. Michael, J. L. Hulley, and G. O. Scott. 2010. A readability assessment of online parkinson’s disease information. The Journal of the Royal College of Physicians of Edinburgh, 40(4):292–296. find
  25. Ikasbil. 2018. Resources. ikasbil.eus. http://www.thesis, University of the Basque Country (UPV/EHU). Advisor: Montse Maritxalar Anglada. ↗
  26. Lilian Kandel and Abraham Moles. 1958. Application de l’indice de flesch à la langue française. Cahiers Etudes de Radio-Télévision, 19:253–274. find
  27. Mohsen Mesgar and Michael Strube. 2015. Graph-based coherence modeling for assessing readability. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, pages 309–318. find
  28. Tapas Kanungo and David Orr. 2009. Predicting the readability of short Web summaries. In Proceedings of the Second ACM International Conference on Web Search and Data Mining, pages 202–211. find
  29. Mohsen Mesgar and Michael Strube. 2016. Lexical coherence graph modeling using word embeddings. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1414–1423. find
  30. Nikolay Karpov, Julia Baranova, and Fedor Vitugin. 2014. Single-sentence readability prediction in russian. In Proceedings of the International Conference on Analysis of Images, Social Networks and Texts, pages 91–100. find
  31. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  32. Mohsen Mesgar and Michael Strube. 2018. A neural local coherence model for text quality assessment. In Proceedings of the 2018 Lexile. 2016. How is the lexile mea-Conference on Empirical Methods in Natural sure of a text determined? Available at: Language Processing, pages 4328–4339. https://lexile.… ↗
  33. Maria Soledad Pera and Yiu-Kai Ng. 2014. Automating readers’ advisory to make book Ion Madrazo Azpiazu. 2017. Towards multilin-recommendations for k-12 readers. In Associ-gual readability assessment. Master’s thesis, ation of Computer Machinery Conference on Boise State University, Boise, Idaho. Rec… find
  34. Ion Madrazo Azpiazu, Nevena Dragovic, Oghenemaro Anuyah, and Maria Soledad Pera. 2018. Looking for the movie Seven or sven from the movie Frozen?: A multi-perspective strategy for recommending queries for children. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval (A… find
  35. Sarah E. Schwarm and Mari Ostendorf. 2005. Reading level assessment using support vector machines and statistical language models. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 523–530.
  36. Seth Spaulding. 1956. A Spanish readability formula. The Modern Language Journal, 40(8):433–441. find
  37. Bangalore S. Manjunath and Wei-Ying Ma. 1996. Texture features for browsing and retrieval of image data. Institute of Electrical and Electronics Engineers Transactions on Pattern Analysis and machine Intelligence, 18(8):837–842. find
  38. Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for Twitter sentiment classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1555–1565.
  39. Ion Madrazo. 2014. Testuen irakurgarritasuna neurtzeko sailkatzaile automatikoa [an automatic classifier of text legibility]. Master’s activations in convolutional network. arXiv preprint arXiv:1505.00853. — [arXiv:1505.00853](https://arxiv.org/abs/1505.00853) find
  40. Jin Wang, Liang-Chih Yu, K. Robert Lai, and Xuejie Zhang. 2016. Dimensional sentiment analysis using a regional CNN-LSTM model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 225–230. find
  41. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015b. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning, pages 2048–2057.
  42. Wikimedia. 2018. Simplification guidelines for simple wikipedia. https://simple.wikipedia.org/wiki/Wikipedia: Simple English Wikipedia. ↗
  43. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 14…
  44. Wizenoze. 2018. Resources. http://www.wizenoze.com. ↗
  45. Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. 2015a. Empirical evaluation of rectified find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/azpiazu-2019-multiattentive-recurrent-neural-network-architecture/