Elamur Research⌕
☰
home / papers / 2023 / ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification

ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification

Paper · 2023 · Taja Kuzman, Igor Mozetič and Nikola Ljubešic

Summary

Labelling web texts by genre is slow and expensive, and even trained linguists often disagree. The authors ask whether ChatGPT, used zero-shot, could replace this manual work for automatic genre identification. They compare it with the X-GENRE classifier, an XLM-RoBERTa model fine-tuned on more than 1,700 hand-labelled texts. Both systems are tested on 100-text samples from the English EN-GINCO dataset and from the Slovenian GINCO test split. All labels are mapped to the 9-category X-GENRE schema. Three setups are evaluated: an English prompt with English texts, an English prompt with Slovenian texts, and a fully Slovenian prompt. On EN-GINCO, which neither system had seen before, ChatGPT outperformed the fine-tuned model. With English instructions, its results on Slovenian were as good as on English. When the prompt itself was in Slovenian, performance fell considerably. The study suggests that, even for smaller languages, large annotation campaigns might shrink to building test sets. It also shows that prompt language and prompt design need further study.

Key points

  • The baseline is the X-GENRE classifier, a fine-tuned XLM-RoBERTa model. It was trained on texts from three annotation campaigns in two languages, using a schema that merges the CORE, FTD and GINCO schemata.
  • The test data were small: 100 instances each.
    • EN-GINCO was sampled from enTenTen20 and was unpublished, so ChatGPT could not have seen it.
    • GINCO comes from the slWaC 2.0 and MaCoCu-sl 1.0 corpora and was originally labelled with 24 categories.
  • The task is hard even for people: inter-annotator agreement on GINCO before adjudication was a nominal Krippendorff's alpha of 0.71. This is above the 0.67 acceptability threshold but below the 0.8 level for good reliability.
  • ChatGPT handled Slovenian texts well when prompted in English, but a fully Slovenian prompt clearly hurt its results.
  • Planned future work includes:
    • larger test sets in more languages via the ChatGPT API;
    • few-shot chain-of-thought prompting;
    • multi-label and span-level genre tagging;
    • using LLM-labelled data to train smaller base-sized models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Jožef Stefan InstituteLjubljana · Slovenia17 works · 2,165 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Taja Kuzman, Igor Mozetič and Nikola Ljubešic. 2023. ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification. arXiv:2303.03953.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.03953 · arXiv 2303.03953

@misc{kuzman2023chatgpt,
  title = {{ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification}},
  author = {Taja Kuzman, Igor Mozetivc and Nikola Ljubevsic},
  year = {2023},
  doi = {10.48550/arxiv.2303.03953},
  eprint = {2303.03953},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.03953}
}

References

34 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky wide web: a collection of very large linguistically processed web-crawled corpora. Language resources and evaluation, 43(3):209–226. find
  2. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. find
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association fo…
  4. Jesse Egbert, Douglas Biber, and Mark Davies. 2015. Developing a bottom-up, user-based method of web register classification. Journal of the Association for Information Science and Technology, 66(9):1817– 1831. find
  5. Tomaž Erjavec and Nikola Ljubešić. 2014. The slWaC 2.0 corpus of the Slovene web. T. Erjavec, J. Žganec Gros (ur.) Jezikovne tehnologije: zbornik, 17:50–55. find
  6. Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications. find
  7. Taja Kuzman, Nikola Ljubešić, and Senja Pollak. 2022a. Assessing Comparability of Genre Datasets via Cross-Lingual and Cross-Dataset Experiments. In Jezikovne tehnologije in digitalna humanistika: zbornik konference, Jezikovne tehnologije in digitalna humanistika: zbornik konference, page 100–107.… find
  8. Taja Kuzman, Peter Rupnik, and Nikola Ljubešić. 2022b. The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild. In Proceedings of the Language Resources and Evaluation Conference, pages 1584–1594, Marseille, France. European Language Resources Association. find
  9. Eugenie Giesbrecht and Stefan Evert. 2009. Is part-of-speech tagging a solved task? An evaluation of POS taggers for the German web as corpus. In Proceedings of the fifth Web as Corpus workshop, pages 27–35. find
  10. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  11. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. arXiv preprint arXiv:2301.07597.
  12. Max Müller-Eberstein, Rob van der Goot, and Barbara Plank. 2021. Genre as Weak Supervision for Cross-lingual Dependency Parsing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4786–4802. find
  13. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. arXiv preprint arXiv:2302.09210. find
  14. OpenAI. 2023. ChatGPT General FAQ. https://help.openai.com/en/articles/ 6783457-chatgpt-general-faq. Accessed: March 3, 2023. ↗
  15. Fan Huang, Haewoon Kwak, and Jisun An. 2023. Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech. arXiv preprint arXiv:2302.07736.
  16. Miloš Jakubíček, Adam Kilgarriff, Vojtěch Kovář, Pavel Rychlỳ, and Vít Suchomel. 2013. The TenTen corpus family. In 7th international corpus linguistics conference CL, pages 125–127. Lancaster University. find
  17. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. pages 4171–4186.
  18. Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popovic… find
  19. Wanda J Orlikowski and JoAnne Yates. 1994. Genre repertoire: The structuring of communicative practices in organizations. Administrative science quarterly, pages 541–574. find
  20. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155.
  21. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a General-Purpose Natural Language Processing Task Solver? arXiv preprint arXiv:2302.06476.
  22. Liina Repo, Valtteri Skantsi, Samuel Rönnqvist, Saara Hellström, Miika Oinonen, Anna Salmela, Douglas Biber, Jesse Egbert, Sampo Pyysalo, and Veronika Laippala. 2021. Beyond the English web: Zero-shot cross-lingual and lightweight monolingual classification of registers. In 16th Conference of the Eu… find
  23. Samuel Rönnqvist, Valtteri Skantsi, Miika Oinonen, and Veronika Laippala. 2021. Multilingual and Zero-Shot is Closing in on Monolingual Web Register Classification. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 157–165. find
  24. Serge Sharoff. 2018. Functional text dimensions for the annotation of web corpora. Corpora, 13(1):65– 95. find
  25. Serge Sharoff. 2021. Genre annotation for the web: text-external and text-internal perspectives. Register studies. find
  26. Serge Sharoff, Zhili Wu, and Katja Markert. 2010. The Web Library of Babel: evaluating genre collections. In LREC. Citeseer. find
  27. Jade Goldstein Stewart and J Callan. 2009. Genre oriented summarization. Ph.D. thesis, Carnegie Mellon University, Language Technologies Institute, School of Computer Science. find
  28. Vít Suchomel. 2020. Genre Annotation of Web Corpora: Scheme and Issues. In Proceedings of the Future Technologies Conference, pages 738–754. Springer. find
  29. Marlies Van der Wees, Arianna Bisazza, and Christof Monz. 2018. Evaluation of machine translation performance across multiple genres and languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  30. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355.
  31. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems.
  32. Bowen Zhang, Daijun Ding, and Liwen Jing. 2022. How would Stance Detection Techniques Evolve after the Launch of ChatGPT? arXiv preprint arXiv:2212.14548.
  33. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT. arXiv preprint arXiv:2302.10198. find
  34. Sven Meyer Zu Eissen and Benno Stein. 2004. Genre classification of web pages. In Annual Conference on Artificial Intelligence, pages 256–269. Springer. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kuzman-2023-chatgpt-beginning-end-manual-linguistic/