Elamur Research⌕
☰
home / papers / 2026 / PUCP-Metrix: An Open-source and Comprehensive Toolkit for Linguistic Analysis of Spanish Texts

PUCP-Metrix: An Open-source and Comprehensive Toolkit for Linguistic Analysis of Spanish Texts

Paper · 2026 · Javier Alonso Villegas Luis and Marco Antonio Sobrevilla Cabezudo

Summary

Spanish has few tools for extracting interpretable linguistic features, which matter for readability, style analysis and explainable NLP. Coh-Metrix-Esp and MultiAzterTest offer limited metric coverage, and MultiAzterTest is also hard to install and slow. The authors built PUCP-Metrix, an open-source Python library on top of spaCy that computes 182 metrics across 13 categories. These include lexical diversity (e.g., MTLD, VOCd), readability formulas adapted for Spanish, syntactic complexity, referential and semantic cohesion, connectives, word frequency, and psycholinguistic norms drawn from EsPal and Stadthagen-Gonzalez et al. The package installs via pip and supports multiprocessing.

The authors tested it on Automated Readability Assessment (ARA) and on Machine-Generated Text Detection (AuTexTification), training classical classifiers on the extracted features:

  • ARA: the features slightly beat MultiAzterTest but trail a fine-tuned RoBERTa-BNE.
  • Detection: the best feature-based model (XGBoost) outperforms both RoBERTa-BNE and the shared task's best reported score, with more balanced results across classes.

The toolkit gives Spanish NLP a scalable, extensible and interpretable feature resource.

Key points

  • Coverage: 182 metrics, compared with 48 in Coh-Metrix-Esp and 141 in MultiAzterTest. Categories missing or underrepresented in earlier tools include semantic cohesion, textual simplicity and psycholinguistics (30 indicators).
  • 2-label ARA: XGBoost on PUCP-Metrix features reaches an overall F1 of 97.46, versus 97.24 with MultiAzter features (not a significant difference) and 98.30 for RoBERTa-BNE.
  • 3-label ARA: the corresponding overall F1 scores are 96.72, 96.56 and 98.13.
  • AuTexTification: XGBoost on PUCP-Metrix features reaches an overall F1 of 71.06. This compares with 70.77 for the shared task's best result, 70.17 for RoBERTa-BNE and 61.63 for MultiAzter features. F1 on human-written texts rises from 42–51 with MultiAzter features to 60–66.
  • Usage and limitations: the package installs with pip install iapucp-metrix and is called through Analyzer.compute_metrics or compute_grouped_metrics. Results depend on spaCy parsing quality and on external lexicons, and the evaluation covers only a few genres.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 worksLiterature and Literary Theory · 519 works

Cite

Cite as Javier Alonso Villegas Luis and Marco Antonio Sobrevilla Cabezudo. 2026. PUCP-Metrix: An Open-source and Comprehensive Toolkit for Linguistic Analysis of Spanish Texts. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-demo.28

@inproceedings{luis2026pucp,
  title = {{PUCP-Metrix: An Open-source and Comprehensive Toolkit for Linguistic Analysis of Spanish Texts}},
  author = {Javier Alonso Villegas Luis and Marco Antonio Sobrevilla Cabezudo},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-demo.28},
  url = {https://doi.org/10.18653/v1/2026.eacl-demo.28}
}

References

31 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John Atkinson and Diego Palma. 2025. An llm-based hybrid approach for enhanced automated essay scoring. Nature: Scientific Reports, 15. find
  2. Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 258–266, Marseille, France. European Language Resources Asso…
  3. Kepa Bengoetxea and Itziar Gonzalez-Dios. 2021. Multiaztertest: A multilingual analyzer on multiple levels of language for readability assessment. arXiv preprint arXiv:2109.04870. — [arXiv:2109.04870](https://arxiv.org/abs/2109.04870) find
  4. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021. MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6974–6996, Onl…
  5. Vittorio Ciccarelli, Cornelia Genz, Nele Mastracchio, Wiebke Petersen, Anna Stein, and Hanxin Xia. 2024. Team art-nat-HHU at SemEval-2024 task 8: Stylistically informed fusion model for MGT-detection. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1690… find
  6. Scott A. Crossley and Kristopher Kyle. 2018. Assessing writing with the tool for the automatic analysis of lexical sophistication (taales). Assessing Writing, 38:46–50. find
  7. Andrew Duchon, Manuel Perea, Nuria Sebastián-Gallés, Antonia Martí, and Manuel Carreiras. 2013. find
  8. Espal: Asier Gutiérrez Fandiño, Jordi Armengol Estapé, Marc Pàmies, Joan Llop Palao, Joaquin Silveira Ocampo, Casimiro Pio Carrino, Carme Armentano Oller, Carlos Rodriguez Penagos, Aitor Gonzalez Agirre, and Marta Villegas. 2022. Maria: Spanish language models. Procesamiento del Lenguaje Natural, 68… find
  9. F Fernández Huerta. 1959. Medidas sencillas de lecturabilidad. Consigna, 214:29–32. find
  10. Itziar Gonzalez-Dios and Kepa Bengoetxea. 2021. Multiaztertest@vaxxstance-iberlef 2021: Identifying stances with language models and linguistic features. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021), volume 2943 of CEUR Workshop Proceedings, pages 192–201. find
  11. Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pa… find
  12. Zhaoyi Hou, Alejandro Ciuba, and Xiang Li. 2025. Improving llm-based automatic essay scoring with linguistic features. In Proceedings of the Innovation and Responsibility in AI-Supported Education Workshop, volume 273 of Proceedings of Machine Learning Research, pages 41–65. PMLR. find
  13. Kevin Jiang. 2016. Douglas biber and bethany gray: Grammatical complexity in academic english: Linguistic change in writing. Applied Linguistics, 37. find
  14. Folkert Kuiken. 2023. Linguistic complexity in second language acquisition. Linguistics Vanguard, 9(s1):83–93. find
  15. Tharindu Kumarage, Joshua Garland, Amrita Bhattacharjee, Kirill Trapeznikov, Scott Ruston, and Huan Liu. 2023. Stylometric detection of ai-generated text in twitter timelines. arXiv preprint arXiv:2303.03697. — [arXiv:2303.03697](https://arxiv.org/abs/2303.03697) find
  16. Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4034–4048, Online. Association for Computational Linguistics.
  17. Sidney Evaldo Leal, Magali Sanches Duran, Carolina Evaristo Scarton, Nathan Siegle Hartmann, and Sandra Maria Aluísio. 2023. Nilc-metrix: assessing the complexity of written and spoken language in brazilian portuguese. Language, Resources and Evaluation, 58(1):73–110. find
  18. Angela Leis, Francesco Ronzano, Miguel A Mayer, Laura I Furlong, and Ferran Sanz. 2019. Detecting signs of depression in tweets in spanish: Behavioral and linguistic analysis. J Med Internet Res, 21(6). find
  19. Fengkai Liu, Tan Jin, and John S. Y. Lee. 2025. Automatic readability assessment for sentences: neural, hybrid and large language models. Language Resources and Evaluation. find
  20. Heinz-Dieter Mass. 1972. Über den zusammenhang zwischen wortschatzumfang und länge eines textes. Zeitschrift für Literaturwissenschaft und Linguistik, 2(8):73. find
  21. Jarvis Scott McCarthy Philip M. 2010. Mtld, vocd-d, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment. In Behavior Research Methods, pages 381–392. find
  22. Danielle S. McNamara, Max M. Louwerse, Philip M. McCarthy, and Arthur C. Graesser. 2010. Coh-metrix: Capturing linguistic features of cohesion. Discourse Processes, 47(4):292–330. find
  23. Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an ai language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050. find
  24. M. Dolores Molina-González, Eugenio Martínez-Cámara, M. Teresa Martín-Valdivia, and L. Alfonso Ureña-López. 2014. Cross-domain sentiment analysis using spanish opinionated words. In Natural Language Processing and Information Systems, pages 214–219, Cham. Springer International Publishing. find
  25. Giovanni Parodi. 2015. Corpus de aprendices de español (caes). Journal of Spanish Language Teaching, 2(2):194–200. find
  26. Kseniia Petukhova, Roman Kazakov, and Ekaterina Kochmar. 2024. PetKaz at SemEval-2024 task 8: Can linguistics capture the specifics of LLM-generated text? In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1140–1147, Mexico City, Mexico. Association for Co… find
  27. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020. MLSUM: The multilingual summarization corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8051–8067, Online. Association for Computational L…
  28. Hans Stadthagen-Gonzalez, Constance Imbault, Miguel A. Pérez Sánchez, and Marc Brysbaert. 2017. Norms of valence and arousal for 14,031 spanish words. Behavior Research Methods, 49(1):111–123. find
  29. Suna-Şeyma Uçar, Itziar Aldabe, Nora Aranberri, and Ana Arruarte. 2024. Exploring automatic readability assessment for science documents within a multilingual educational context. International Journal of Artificial Intelligence in Education, 34(4):1417– 1459. find
  30. Laura Vásquez-Rodríguez, Pedro-Manuel Cuenca-Jiménez, Sergio Morales-Esquivel, and Fernando Alva-Manchego. 2022. A benchmark for neural readability assessment of texts in Spanish. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), pages 188–198, Abu Dh… find
  31. Jinshan Zeng, Xianchao Tong, Xianglong Yu, Wenyan Xiao, and Qing Huang. 2024. Interpretara: Enhancing hybrid automatic readability assessment with linguistic feature interpreter and contrastive learning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19497–19505. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luis-2026-pucp-metrix-open-source-comprehensive/