Elamur Research⌕
☰
home / papers / 2024 / Beyond Error Categories: A Contextual Approach of Evaluating Emerging Spell and Grammar Checkers

Beyond Error Categories: A Contextual Approach of Evaluating Emerging Spell and Grammar Checkers

Summary

The authors propose a new way of building test data for evaluating spelling and grammar checking systems, including those based on large language models (LLMs). Such datasets usually consist of individual sentences with predefined error categories. Here, by contrast, whole texts are used, called error documents. This makes it possible to test whether a system takes context and world knowledge into account. For Icelandic, the texts were selected from the Icelandic Gigaword Corpus and the Icelandic Common Crawl Corpus. Most of the errors are natural, and artificially introduced ones are rare. The dataset has three parts with different annotation: one marks only the error spans, another corrects the entire text, and the most complete one contains error marking, corrections, plain-language explanations and an assessment of error severity. The main novelty lies precisely in the explanations and the severity scale. They make it possible to evaluate LLMs that not only correct a text but also explain to the user how to write better. The result is an openly licensed test set on which different proofreading tools can be compared. This is especially useful for low-resource languages.

Key points

  • The total size is about 380 thousand words and over 9,000 annotations. Type 1 has almost 200 thousand words and ~3,300 annotations, type 2 slightly under 150 thousand words and ~5,000 annotations, and type 3 ~30 thousand words and ~900 annotations.
  • Type 1 is suitable only for evaluating how accurately a system detects errors. Type 2 also allows evaluating the accuracy of corrections and computing GLEU, and the error spans in it are determined automatically from the difference between text versions. Type 3 covers all of this, as well as explanations.
  • Error severity was rated on a scale from 1 to 5, where 5 denotes the most serious error. The rating shows how much the error could damage the reputation of the text's author.
  • Only unambiguous errors were corrected; stylistic ones were left untouched. The annotators followed the spelling norms of the Icelandic Language Council. The annotation was done by three native speakers of Icelandic with a background in philology or experience in professional proofreading.
  • The dataset is published in CLARIN-IS as "Grammatical error correction test set" (Símonarson et al., 2023). The Common Crawl texts are released together with their originals. The Gigaword originals are not published, as their licenses do not permit it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{arnardottir2024beyond,
  title = {{Beyond Error Categories: A Contextual Approach of Evaluating Emerging Spell and Grammar Checkers}},
  author = {Arnardóttir, THórunn and Ingólfsdóttir, Svanhvít Lilja and Símonarson, Haukur Barri and Einarsson, Hafsteinn and Ingason, Anton Karl and THorsteinsson, Vilhjálmur},
  booktitle = {Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024},
  year = {2024},
  doi = {10.63317/452t2ju9vgnk},
  url = {https://doi.org/10.63317/452t2ju9vgnk}
}

References

42 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Þórunn Arnardóttir, Isidora Glisic, Annika Simonsen, Lilja Stefánsdóttir, and Anton Ingason. 2022. Error corpora for different informant groups: Annotating and analyzing texts from L2 speakers, people with dyslexia and children. In Proceedings of the 19th International Conference on Natural Language… find
  2. Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. 2023. Is ChatGPT a highly fluent grammatical error correction system? A comprehensive evaluation. arXiv preprint arXiv:2304.01746. find
  3. Þórunn Arnardóttir, Xindan Xu, Dagbjört Guðmundsdóttir, Lilja Björk Stefánsdóttir, and Anton Karl Ingason. 2021. Creating an error corpus: Annotation and applicability. In CLARIN Annual Conference Proceedings, pages 59–63. find
  4. Luciano Floridi and Massimo Chiriatti. 2020. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681–694. find
  5. Starkaður Barkarson, Steinþór Steingrímsson, and Hildur Hafsteinsdóttir. 2022. Evolving large text corpora: Four versions of the Icelandic Gigaword corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2371–2381, Marseille, France. European Language Resources A… find
  6. Nizar Habash and David Palfreyman. 2022. ZAE-BUC: An annotated Arabic-English bilingual writer corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79–88, Marseille, France. European Language Resources Association. find
  7. Marie Bexte, Ronja Laarmann-Quante, Andrea Horbach, and Torsten Zesch. 2022. LeSpell - a multi-lingual benchmark corpus of spelling errors to develop spellchecking methods for learner language. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 697–706, Marseille, F… find
  8. Svanhvít Lilja Ingólfsdóttir, Pétur Ragnarsson, Haukur Jónsson, Haukur Símonarson, Vilhjálmur Þorsteinsson, and Vésteinn Snæbjarnarson. 2023. Byte-level grammatical error correction using synthetic and curated corpora. In Proceedings of the 61st Annual Meeting of the Association for Computational Li… find
  9. Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018. doccano: Text annotation tool for human. Software available from https://github.com/doccano/ doccano. ↗
  10. Jianshu Ji, Qinlong Wang, Kristina Toutanova, Yongen Gong, Steven Truong, and Jianfeng Gao. 2017. A nested attention neural hybrid model for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 753–762… find
  11. Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volum… find
  12. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. Approaching neural grammatical error correction as a low-resource machine translation task. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  13. Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2016. GLEU without tuning. arXiv preprint arXiv:1605.02592. find
  14. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–14, Baltimore, Ma… find
  15. Anisia Katinskaia, Maria Lebedeva, Jue Hou, and Roman Yangarber. 2022. Semi-automatically annotated learner corpus for Russian. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 832–839, Marseille, France. European Language Resources Association. find
  16. Anna Nikulásdóttir, Þórunn Arnardóttir, Starkaður Barkarson, Jón Guðnason, Þorsteinn Gun-narsson, Anton Ingason, Haukur Jónsson, Hrafn Loftsson, Hulda Óladóttir, Eiríkur Rögnvaldsson, Einar Sigurðsson, Atli Sigurgeirsson, Vésteinn Snæbjarnarson, Steinþór Steingrímsson, and Gunnar Örnólfsson. 2022. H… find
  17. Katerina Korre and John Pavlopoulos. 2022. Enriching grammatical error correction resources for Modern Greek. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4984–4991, Marseille, France. European Language Resources Association. find
  18. Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications. find
  19. Hulda Óladóttir, Þórunn Arnardóttir, Anton Ingason, and Vilhjálmur Þorsteinsson. 2022. Developing a spell and grammar checker for Icelandic using an error corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4644–4653, Marseille, France. European Language Reso… find
  20. Yinghui Li, Haojing Huang, Shirong Ma, Yong Jiang, Yangning Li, Feng Zhou, Hai-Tao On the Zheng, and Qingyu Zhou. 2023. (in)effectiveness of large language models arXiv preprint for Chinese text correction. arXiv:2307.09007. — [arXiv:2307.09007](https://arxiv.org/abs/2307.09007) find
  21. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… find
  22. Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Da… find our card
  23. Maria Carolina Penteado and Fábio Perez. 2023. Evaluating GPT-3.5 and GPT-4 on grammatical error correction for Brazilian Portuguese. arXiv preprint arXiv:2306.15788. find
  24. Fanyi Qu and Yunfang Wu. 2023. Evaluating the capability of large-scale language models on Chinese grammatical error correction task. arXiv preprint arXiv:2307.03972. find
  25. Georg Rehm and Andy Way, editors. 2023. European Language Equality - A Strategic Agenda for Digital Language Equality. Cognitive Technologies. Springer. find
  26. Vésteinn Snæbjarnarson, Haukur Barri Símonarson, Pétur Orri Ragnarsson, Svanhvít Lilja Ingólfsdóttir, Haukur Jónsson, Vilhjálmur Þorsteinsson, and Hafsteinn Einarsson. 2022. A warm start and a clean crawled corpus - a recipe for good language models. In Proceedings of the Thirteenth Language Resourc… find
  27. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. brat: a web-based tool for NLP-assisted text annotation. In Proceedings of the \#\# 8. Language Resource References Demonstrations at the 13th Conference of the European Chapter of the Association f… find
  28. Starkaður Barkarson and Steinþór Steingrímsson. 2022. IGC-news2-22.10 (annotated version). CLARIN-IS. find
  29. Yujin Takahashi, Masahiro Kaneko, Masato Mita, and Mamoru Komachi. 2022. ProQE: Proficiency-wise quality estimation dataset for grammatical error correction. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5994– 6000, Marseille, France. European Language Resource… find
  30. Björn Halldórsson, Árni Davíð Magnússon, Finnur Ágúst Ingimundarson, Einar Freyr Sigurðsson, Steinþór Steingrímsson, Halldóra Jónsdóttir, and Þórdís Úlfarsdóttir. 2022. Idiomatic expressions (Icelandic and English) 22.09. CLARIN-IS. find
  31. Anton Karl Ingason, Þórunn Arnardóttir, Lilja Björk Stefánsdóttir, and Xindan Xu. 2021a. The Icelandic child language error corpus (IceCLEC) version 1.1. CLARIN-IS. find
  32. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv pr… find
  33. Anton Karl Ingason, Þórunn Arnardóttir, Lilja Björk Stefánsdóttir, Xindan Xu, Dagbjört Guðmundsdóttir, and Isidora Glišić. 2022a. The Icelandic dyslexia error corpus 1.2 (22.10). CLARIN-IS. find
  34. Anton Karl Ingason, Lilja Björk Stefánsdóttir, Þórunn Arnardóttir, and Xindan Xu. 2021b. Icelandic error corpus (IceEC) version 1.1. CLARIN-IS. find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017a. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  36. Anton Karl Ingason, Lilja Björk Stefánsdóttir, Þórunn Arnardóttir, Xindan Xu, Isidora Glišić, and Dagbjört Guðmundsdóttir. 2022b. The Icelandic L2 error corpus (IceL2EC) 1.3 (22.10). CLARIN-IS. find
  37. Baoxin Wang, Xingyi Duan, Dayong Wu, Wanxiang Che, Zhigang Chen, and Guoping Hu. 2022. CCTC: A cross-sentence Chinese text correction dataset for native speakers. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3331– 3341, Gyeongju, Republic of Korea. Internat… find
  38. Miðeind. 2022. Icelandic common crawl corpus (IC3). Hugging Face. find
  39. Haukur Barri Símonarson, Svanhvít Lilja Ingólfsdóttir, Þórunn Arnardóttir, Dagbjört Guðmundsdóttir, Ella María Georgsdóttir, and Guðrún Lilja Friðjónsdóttir. 2023. Grammatical error correction test set. CLARIN-IS. find
  40. Yu Wang, Yuelin Wang, Jie Liu, and Zhuo Liu. 2020. A comprehensive survey of grammar error correction. arXiv preprint arXiv:2005.06600. — [arXiv:2005.06600](https://arxiv.org/abs/2005.06600) find
  41. Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023. ChatGPT or Grammarly? evaluating ChatGPT on grammatical error correction benchmark. arXiv preprint arXiv:2303.13648. find
  42. Zheng Yuan and Ted Briscoe. 2016. Grammatical error correction using neural machine translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 380–386, San Diego, California. Association for Co… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/arnardottir-2024-beyond-error-categories-contextual-approach/