Elamur Research⌕
☰
home / papers / 2023 / ChatGPT: Jack of all trades, master of none

ChatGPT: Jack of all trades, master of none

Summary

The authors, from Wrocław University of Science and Technology, ask how well ChatGPT handles classic analytical NLP problems compared with specialised state-of-the-art (SOTA) models. Earlier evaluations were mostly manual and small. This study builds an automated prompting pipeline and applies it to 25 public datasets. Many of the tasks are subjective even for human annotators, such as sentiment, emotion, offensiveness, humor and stance. Others call for more objective reasoning, for example determining which meaning a word carries in context, judging whether a sentence is linguistically acceptable, and answering questions. GPT-4 was also run on five task subsets.

Across zero-shot and few-shot settings, ChatGPT typically falls about a quarter below SOTA. The gap grows as tasks get harder, and pragmatic problems like emotion recognition suffer most. GPT-4 narrows the gap on semantic tasks. Adding a few annotated examples from a specific user improves user-level predictions on subjective tasks. A qualitative review uncovers refusals and biases that likely stem from OpenAI's trainer guidelines. The work questions whether conversational fluency signals real usefulness and calls for better validation practices.

Key points

  • The study covers 25 diverse NLP datasets and over 49k automatically collected ChatGPT and GPT-4 responses.
  • On average, ChatGPT loses about 25% in quality relative to SOTA, in both zero-shot and few-shot settings.
  • The loss correlates with task difficulty (lower SOTA scores mean larger drops) and peaks on pragmatic tasks such as emotion recognition.
  • GPT-4, tested on five subsets, shows a markedly smaller loss than ChatGPT on semantic tasks.
  • Random Contextual Few-Shot Personalization significantly improves user-based predictions. Qualitative cases include an over-strict content-policy refusal on a PolEmo2.0 hostel review.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

107 papers of our library cite it (of the 140884 whose bibliographies we read); 16 of them have a page here, 91 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1016/j.inffus.2023.101861 · arXiv 2302.10724

@misc{kocon2023chatgpt,
  title = {{ChatGPT: Jack of all trades, master of none}},
  author = {Kocon, Jan and Cichecki, Igor and Kaszyca, Oliwier and Kochanek, Mateusz and Szydlo, Dominika and Baran, Joanna and Bielaniewicz, Julita and Gruza, Marcin and Janz, Arkadiusz and Kanclerz, Kamil and Kocon, Anna and Koptyra, Bartlomiej and Mieleszczenko-Kowszewicz, Wiktoria and Milkowski, Piotr and Oleksy, Marcin and Piasecki, Maciej and Radlinski, Lukasz and Wojtasik, Konrad and Wozniak, Stanislaw and Kazienko, Przemyslaw},
  year = {2023},
  doi = {10.1016/j.inffus.2023.101861},
  eprint = {2302.10724},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1016/j.inffus.2023.101861}
}

References

106 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, I. Polosukhin, Attention is all you need, in: I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems, volume 30, Curran Asso… find
  2. J. Ni, T. Young, V. Pandelea, F. Xue, E. Cambria, Recent advances in deep learning based dialogue systems: a systematic survey, in: Artif. Intell. Rev., volume 56, 2023, pp. 3055–3155. URL: https://doi.org/10.1007/s10462-022-10248-8 . doi: 10.1007/s10462-022-10248-8 . — [doi:10.1007/s10462-022-10248-8](https://doi.org/10.1007/s10462-022-10248-8) ↗
  3. T. Lin, Y. Wang, X. Liu, X. Qiu, A survey of transformers, in: AI Open, volume 3, 2022, pp. 111–132. URL: https://www.sciencedirect.com/science/article/pii/S2666651022000146 . doi: https://doi.org/10.1016/j.aiopen.2022.10.001 . — [doi:10.1016/j.aiopen.2022.10.001](https://doi.org/10.1016/j.aiopen.2022.10.001) ↗
  4. R. Johnson, T. Zhang, Supervised and semi-supervised text categorization using LSTM for region embeddings, in: M. Balcan, K. Q. Weinberger (Eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and… find
  5. W. Liu, Z. Wang, X. Liu, N. Zeng, Y. Liu, F. E. Alsaadi, A survey of deep neural network architectures and their applications, in: Neurocomputing, volume 234, Elsevier, 2017, pp. 11–26. URL: https://www.sciencedirect.com/science/article/pii/ S0925231216315533 . doi: https://doi.org/10.1016/j.neucom.… — [doi:10.1016/j.neucom.2016.12](https://doi.org/10.1016/j.neucom.2016.12) ↗
  6. B. Alshemali, J. Kalita, Improving the reliability of deep neural networks in NLP: A review, in: Knowl. Based Syst., volume 191, 2020, p. 105210. URL: https://doi.org/10.1016/j.knosys. 2019.105210 . doi: 10.1016/j.knosys.2019.105210 . — [doi:10.1016/j.knosys](https://doi.org/10.1016/j.knosys) ↗
  7. G. Liu, J. Guo, Bidirectional LSTM with attention mechanism and convolutional layer for text classification, in: Neurocomputing, volume 337, 2019, pp. 325–338. URL: https://doi.org/10.1016/j. neucom.2019.01.078 . doi: 10.1016/j.neucom.2019.01.078 . — [doi:10.1016/j](https://doi.org/10.1016/j) find
  8. Z. C. Lipton, A critical review of recurrent neural networks for sequence learning, in: arXiv, volume abs/1506.00019, 2015. URL: http://arxiv.org/abs/1506.00019 . arXiv:1506.00019 . — [arXiv:1506.00019](https://arxiv.org/abs/1506.00019) ↗
  9. A. Gillioz, J. Casas, E. Mugellini, O. A. Khaled, Overview of the transformer-based models for NLP tasks, in: M. Ganzha, L. A. Maciaszek, M. Paprzycki (Eds.), Proceedings of the 2020 Federated Conference on Computer Science and Information Systems, FedCSIS 2020, Sofia, Bulgaria, September 6-9, 2020,… — [doi:10.15439/2020f20](https://doi.org/10.15439/2020f20) find
  10. W. Rahman, M. K. Hasan, S. Lee, A. B. Zadeh, C. Mao, L. Morency, M. E. Hoque, Integrating multimodal information in large pre-trained transformers, in: D. Jurafsky, J. Chai, N. Schluter, J. R. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,… find our card
  11. A. V. Ganesan, M. Matero, A. R. Ravula, H. Vu, H. A. Schwartz, Empirical evaluation of pre-trained transformers for human-level NLP: the role of sample size and dimensionality, in: K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tür, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, Y. Zh… find
  12. A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al., Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, in: arXiv, volume abs/2206.04615, 2022. URL: https://doi.org/10.48550/arXi… ↗ our card
  13. J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
  14. P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, L. Kaiser, N. Shazeer, Generating wikipedia by summarizing long sequences, in: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, OpenReview.net, 2018.… find our card
  15. A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al., Improving language understanding by generative pre-training, OpenAI, 2018. find
  16. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language Models are Unsupervised Multitask Learners, OpenAI, 2019. find
  17. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, in: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020,… find
  18. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., Training language models to follow instructions with human feedback, in: arXiv, volume abs/2203.02155, 2022. URL: https://doi.org/10.48550/arXiv.2203.02155 . doi: 10.48550/arXiv.220… ↗
  19. OpenAI, Gpt-4 technical report, 2023. arXiv:2303.08774 . find our card
  20. J. Kocoń, A. Figas, M. Gruza, D. Puchalska, T. Kajdanowicz, P. Kazienko, Offensive, aggressive, and hate speech analysis: From data-centric to human-centered approach, in: Information Processing & Management, volume 58, 2021, p. 102643. URL: https://www.sciencedirect.com/science/article/pii/ S030645… — [doi:10.1016/j.ipm.2021.102643](https://doi.org/10.1016/j.ipm.2021.102643) ↗
  21. K. Kanclerz, A. Figas, M. Gruza, T. Kajdanowicz, J. Kocon, D. Puchalska, P. Kazienko, Controversy and conformity: from generalized to personalized aggressiveness detection, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Co… find
  22. P. Kazienko, J. Bielaniewicz, M. Gruza, K. Kanclerz, K. Karanowski, P. Miłkowski, J. Kocoń, Human-centered neural reasoning for subjective content processing: Hate speech, emotions, and humor, in: Information Fusion, volume 94, 2023, pp. 43–65. URL: https://www.sciencedirect.com/science/article/pii/… — [doi:10.1016/j.inffus.2023.01](https://doi.org/10.1016/j.inffus.2023.01) ↗
  23. P. Schramowski, C. Turan, N. Andersen, C. A. Rothkopf, K. Kersting, Large pre-trained language models contain human-like biases of what is right and wrong to do, Nature Machine Intelligence 4 (2022) 258–268. find
  24. E. Ferrara, Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language models, arXiv preprint arXiv:2304.03738 (2023). — [arXiv:2304.03738](https://arxiv.org/abs/2304.03738) find
  25. T. Susnjak, ChatGPT: The end of online exam integrity?, in: arXiv, volume abs/2212.09292, 2022. URL: https://doi.org/10.48550/ arXiv.2212.09292 . doi: 10.48550/arXiv.2212.09292 . arXiv:2212.09292 . — [doi:10.48550/arxiv.2212.09292](https://doi.org/10.48550/arxiv.2212.09292) find
  26. T. H. Kung, M. Cheatham, A. Medinilla, ChatGPT, C. Sillos, L. De Leon, C. Elepano, M. Madriaga, R. Aggabao, G. Diaz-Candido, et al., Performance of ChatGPT on usmle: Potential for ai-assisted medical education using large language models, medRxiv (2022) 2022–12. find
  27. B. Lund, W. Ting, Chatting about ChatGPT: How May AI and GPT Impact Academia and Libraries?, Lund, BD, & Wang (2023). find
  28. F. Antaki, S. Touma, D. Milad, J. El-Khoury, R. Duval, Evaluating the performance of ChatGPT in ophthalmology: An analysis of its successes and shortcomings, in: medRxiv, Cold Spring Harbor Laboratory Press, 2023. URL: https://www.medrxiv.org/content/early/ 2023/01/26/2023.01.22.23284882 . doi: 10.1… — [doi:10.1101/2023.01.22.23284882](https://doi.org/10.1101/2023.01.22.23284882) ↗
  29. A. M. Perlman, et al., The implications of openai’s assistant for legal services and society, Available at SSRN (2022). find
  30. T. Goyal, J. J. Li, G. Durrett, News summarization and evaluation in the era of gpt-3, 2022. URL: https://arxiv.org/abs/2209.12356 . doi: 10.48550/ARXIV.2209.12356 . ↗
  31. L. Zhao, W. Alhoshan, A. Ferrari, K. J. Letsholo, Classification of natural language processing techniques for requirements engineering, in: arXiv, volume abs/2204.04282, 2022. URL: https://doi.org/10.48550/arXiv.2204.04282 . doi: 10.48550/arXiv.2204.04282 . arXiv:2204.04282 . — [doi:10.48550/arxiv.2204.04282](https://doi.org/10.48550/arxiv.2204.04282) ↗
  32. T. Ganegedara, Natural Language Processing with TensorFlow: Teach language to machines using Python’s deep learning library, Packt Publishing Ltd, 2018. find
  33. B. Guo, X. Zhang, Z. Wang, M. Jiang, J. Nie, Y. Ding, J. Yue, Y. Wu, How close is ChatGPT to human experts? comparison corpus, evaluation, and detection, 2023. URL: https://arxiv.org/abs/2301.07597 . doi: 10.48550/ARXIV.2301.07597 . ↗ our card
  34. A. Gilson, C. Safranek, T. Huang, V. Socrates, L. Chi, R. A. Taylor, D. Chartash, How does ChatGPT perform on the medical licensing exams? the implications of large language models for medical education and knowledge assessment, in: medRxiv, Cold Spring Harbor Laboratory Press, 2022. URL: https://w… — [doi:10.1101/2022.12.23.22283901](https://doi.org/10.1101/2022.12.23.22283901) find
  35. K. Wenzlaff, S. Spaeth, Smarter than humans? validating how openai’s ChatGPT model explains crowdfunding, alternative finance and community finance., Validating how OpenAI’s ChatGPT model explains Crowdfunding, Alternative Finance and Community Finance.(December 22, 2022) (2022). find
  36. T. Phillips, A. Saleh, K. D. Glazewski, C. E. Hmelo-Silver, B. Mott, J. C. Lester, Exploring the use of gpt-3 as a tool for evaluating text-based collaborative discourse, Examining pedagogical data literacy: results of a survey among school teachers at upper secondary level in Switzerland (2022) 54. find
  37. C. A. Gao, F. M. Howard, N. S. Markov, E. C. Dyer, S. Ramesh, Y. Luo, A. T. Pearson, Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers, in: bioRxiv, Cold Spring Harbor Laborator… — [doi:10.1101/2022.12.23.521610](https://doi.org/10.1101/2022.12.23.521610) find
  38. Ö. Aydın, E. Karaarslan, Openai ChatGPT generated literature review: Digital twin in healthcare, Available at SSRN 4308687 (2022). find
  39. K. Jeblick, B. Schachtner, J. Dexl, A. Mittermeier, A. T. Stüber, J. Topalis, T. Weber, P. Wesp, B. Sabel, J. Ricke, M. Ingrisch, ChatGPT makes medicine easy to swallow: An exploratory case study on simplified radiology reports, 2022. URL: https://arxiv.org/abs/2212.14882 . doi: 10.48550/ARXIV.2212… — [doi:10.48550/arxiv.2212.14882](https://doi.org/10.48550/arxiv.2212.14882) ↗
  40. Y. Chen, S. Eger, Transformers go for the lols: Generating (humourous) titles from scientific abstracts end-to-end, in: arXiv, volume abs/2212.10522, 2022. URL: https://doi.org/10.48550/arXiv. 2212.10522 . doi: 10.48550/arXiv.2212.10522 . arXiv:2212.10522 . ↗
  41. W. Jiao, W. Wang, J.-t. Huang, X. Wang, Z. Tu, Is ChatGPT a good translator? a preliminary study, 2023. URL: https://arxiv.org/abs/ 2301.08745 . doi: 10.48550/ARXIV.2301.08745 . ↗ our card
  42. W. Tabone, J. de Winter, Using ChatGPT for Human-Computer Interaction Research: A Primer, 2023. find
  43. B. Kutela, K. Msechu, S. Das, E. Kidando, ChatGPT’s scientific writings: A case study on traffic safety, Available at SSRN 4329120 (2023). find
  44. R. Karanjai, Targeted phishing campaigns using large scale language models, in: arXiv, volume abs/2301.00665, 2023. URL: https://doi.org/10.48550/arXiv.2301.00665 . doi: 10.48550/arXiv.2301.00665 . arXiv:2301.00665 . — [doi:10.48550/arxiv.2301.00665](https://doi.org/10.48550/arxiv.2301.00665) ↗
  45. A. Azaria, ChatGPT Usage and Limitations, 2022. URL: https://hal.science/hal-03913837 , working paper or preprint. ↗
  46. M. M. Amin, E. Cambria, B. W. Schuller, Will affective computing emerge from foundation models and general ai? a first evaluation on ChatGPT, 2023. arXiv:2303.03186 . — [arXiv:2303.03186](https://arxiv.org/abs/2303.03186) find
  47. Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, Q. V. Do, Y. Xu, P. Fung, A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity, 2023. arXiv:2302.04023 . find our card
  48. W. Castillo-González, The importance of human supervision in the use of ChatGPT as a support tool in scientific writing, Metaverse Basic and Applied Research 2 (2022) 29. find
  49. I. E. Karfi, S. E. Fkihi, An ensemble of arabic transformer-based models for arabic sentiment analysis, International Journal of Advanced Computer Science and Applications 13 (2022). find
  50. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. Or… find
  51. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. Or… find
  52. T. Y. Zhuo, Y. Huang, C. Chen, Z. Xing, Exploring AI ethics of ChatGPT: A diagnostic analysis, in: arXiv, volume abs/2301.12867, 2023. URL: https://doi.org/10.48550/arXiv.2301.12867 . doi: 10. 48550/arXiv.2301.12867 . arXiv:2301.12867 . ↗ our card
  53. J. Wang, X. Hu, W. Hou, H. Chen, R. Zheng, Y. Wang, L. Yang, H. Huang, W. Ye, X. Geng, B. Jiao, Y. Zhang, X. Xie, On the robustness of ChatGPT: An adversarial and out-of-distribution perspective, 2023. arXiv:2302.12095 . find
  54. B. Peng, C. Li, P. He, M. Galley, J. Gao, Instruction tuning with gpt-4, 2023. arXiv:2304.03277 . find our card
  55. H. Nori, N. King, S. M. McKinney, D. Carignan, E. Horvitz, Capabilities of gpt-4 on medical challenge problems, 2023. arXiv:2303.13375 . find our card
  56. M. B. I. au2, D. M. Katz, Gpt takes the bar exam, 2022. arXiv:2212.14402 . — [arXiv:2212.14402](https://arxiv.org/abs/2212.14402) find
  57. M. Kosinski, Theory of mind may have spontaneously emerged in large language models, 2023. arXiv:2302.02083 . find our card
  58. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, G. Neubig, Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, in: ACM Comput. Surv., volume 55, 2023, pp. 195:1–195:35. URL: https://doi.org/10.1145/3560815 . doi: 10.1145/3560815 . — [doi:10.1145/3560815](https://doi.org/10.1145/3560815) ↗
  59. C. W. Morris, Foundations of the theory of signs, in: International encyclopedia of unified science, Chicago University Press, 1938, pp. 1–59. find
  60. S. Blum-Kulka, M. Hamo, Discourse pragmatics, Discourse studies: A multidisciplinary introduction 2 (2011) 143–164. find
  61. J. Firth, A Synopsis of Linguistic Theory 1930-1955, Philological Society, Oxford, 1957. Reprinted in Palmer, F. (ed. 1968) Selected Papers of J. R. Firth, Longman, Harlow. find
  62. E. Wulczyn, N. Thain, L. Dixon, Ex machina: Personal attacks seen at scale, in: R. Barrett, R. Cummings, E. Agichtein, E. Gabrilovich (Eds.), Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, ACM, 2017, pp. 1391–1399. URL: https://doi.or… — [doi:10.1145/3038912.3052591](https://doi.org/10.1145/3038912.3052591) ↗
  63. I. D. Kivlichan, Z. Lin, J. Z. Liu, L. Vasserman, Measuring and improving model-moderator collaboration using uncertainty estimation, in: arXiv, volume abs/2107.04212, 2021. URL: https://arxiv.org/abs/2107.04212 . arXiv:2107.04212 . ↗
  64. A. Warstadt, A. Singh, S. R. Bowman, Neural network acceptability judgments, in: Trans. Assoc. Comput. Linguistics, volume 7, 2019, pp. 625–641. URL: https://doi.org/10.1162/tacl_a_00290 . doi: 10. 1162/tacl\\_a\\_00290 . ↗
  65. S. Wang, H. Fang, M. Khabsa, H. Mao, H. Ma, Entailment as few-shot learner, in: arXiv, volume abs/2104.14690, 2021. URL: https://arxiv.org/abs/2104.14690 . arXiv:2104.14690 . — [arXiv:2104.14690](https://arxiv.org/abs/2104.14690) ↗
  66. I. Annamoradnejad, Colbert: Using BERT sentence embedding for humor detection, in: arXiv, volume abs/2004.12765, 2020. URL: https://arxiv.org/abs/2004.12765 . arXiv:2004.12765 . — [arXiv:2004.12765](https://arxiv.org/abs/2004.12765) ↗
  67. R. Siddiqui, SARCASMANIA: Sarcasm Exposed!, http://www.kaggle.com/rmsharks4/sarcasmania-dataset , 2019. [Online; accessed 02-February-2023]. ↗
  68. P. Kumar, G. Sarin, WELMSD - word embedding and language model based sarcasm detection, in: Online Inf. Rev., volume 46, 2022, pp. 1242–1256. URL: https://doi.org/10.1108/ OIR-03-2021-0184 . doi: 10.1108/OIR-03-2021-0184 . — [doi:10.1108/oir-03-2021-0184](https://doi.org/10.1108/oir-03-2021-0184) find
  69. J. M. G. Hidalgo, T. A. Almeida, A. Yamakami, On the validity of a new SMS spam collection, in: 11th International Conference on Machine Learning and Applications, ICMLA, Boca Raton, FL, USA, December 12-15, 2012. Volume 2, volume 2, IEEE, IEEE, 2012, pp. 240–245. URL: https://doi.org/10.1109/ICMLA.… — [doi:10.1109/icmla.2012.211](https://doi.org/10.1109/icmla.2012.211) ↗
  70. T. Sahmoud, M. Mikki, Spam detection using BERT, in: arXiv, volume abs/2206.02443, 2022. URL: https://doi.org/10.48550/arXiv. 2206.02443 . doi: 10.48550/arXiv.2206.02443 . arXiv:2206.02443 . — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) ↗
  71. M. T. Pilehvar, J. Camacho-Collados, Wic: the word-in-context dataset for evaluating context-sensitive meaning representations, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find our card
  72. B. Zoph, I. Bello, S. Kumar, N. Du, Y. Huang, J. Dean, N. Shazeer, W. Fedus, St-moe: Designing stable and transferable sparse expert models, 2022. URL: https://arxiv.org/abs/2202.08906 . doi: 10. 48550/ARXIV.2202.08906 . ↗
  73. A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, S. Bowman, Superglue: A stickier benchmark for general-purpose language understanding systems, in: H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, R. Garnett (Eds.), Advances in Neural Information Proces… find our card
  74. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, S. Bowman, GLUE: A multi-task benchmark and analysis platform for natural language understanding, in: Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Association for Computational Linguistics, Br… find our card
  75. B. Patra, S. Singhal, S. Huang, Z. Chi, L. Dong, F. Wei, V. Chaudhary, X. Song, Beyond english-centric bitexts for better multilingual language representation learning, in: arXiv, volume abs/2210.14867, 2022. URL: https://doi.org/10.48550/arXiv.2210. 14867 . doi: 10.48550/arXiv.2210.14867 . arXiv:22… ↗
  76. P. Rajpurkar, R. Jia, P. Liang, Know what you don’t know: Unanswerable questions for squad, in: arXiv, volume abs/1806.03822, 2018. URL: http://arxiv.org/abs/1806.03822 . arXiv:1806.03822 . ↗ our card
  77. P. He, J. Gao, W. Chen, Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, in: arXiv, volume abs/2111.09543, 2021. URL: https://arxiv.org/abs/ 2111.09543 . arXiv:2111.09543 . ↗ our card
  78. K. Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, J. Schulman, Training verifiers to solve math word problems, in: arXiv, volume abs/2110.14168, 2021. URL: https://arxiv.org/abs/ 2110.14168 . arXiv:2110.14168 . ↗ our card
  79. Y. Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J. Lou, W. Chen, On the advance of making language models better reasoners, in: arXiv, volume abs/2206.02336, 2022. URL: https://doi.org/10.48550/arXiv. 2206.02336 . doi: 10.48550/arXiv.2206.02336 . arXiv:2206.02336 . — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) ↗
  80. D. Demszky, D. Movshovitz-Attias, J. Ko, A. S. Cowen, G. Nemade, S. Ravi, Goemotions: A dataset of fine-grained emotions, in: D. Jurafsky, J. Chai, N. Schluter, J. R. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-1… find
  81. A. Ngo, A. Candri, T. Ferdinan, J. Kocon, W. Korczynski, StudEmo: A non-aggregated review dataset for personalized emotion recognition, in: Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022, European Language Resources Association, Marseille, France, 2022, pp. 46–55. URL:… find
  82. I. Price, J. Gifford-Moore, J. Flemming, S. Musker, M. Roichman, G. Sylvain, N. Thain, L. Dixon, J. Sorensen, Six attributes of unhealthy conversations, in: Proceedings of the Fourth Workshop on Online Abuse and Harms, Association for Computational Linguistics, Online, 2020, pp. 114–124. URL: https:… find
  83. J. Kocon, P. Milkowski, M. Zasko-Zielinska, Multi-level sentiment analysis of polemo 2.0: Extended corpus of multi-domain consumer reviews, in: M. Bansal, A. Villavicencio (Eds.), Proceedings of the 23rd Conference on Computational Natural Language Learning, CoNLL 2019, Hong Kong, China, November 3-… find
  84. F. Barbieri, J. Camacho-Collados, L. Espinosa Anke, L. Neves, TweetEval: Unified benchmark and comparative evaluation for tweet classification, in: Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, 2020, pp. 1644–1650. URL: http… find our card
  85. D. Loureiro, F. Barbieri, L. Neves, L. Espinosa Anke, J. Camacho-collados, TimeLMs: Diachronic language models from Twitter, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Association for Computational Linguistics, Dublin, Ireland,… find our card
  86. Y. Xu, J. Liu, J. Gao, Y. Shen, X. Liu, Towards human-level machine reading comprehension: Reasoning and inference with multiple strategies, in: arXiv, volume abs/1711.04964, 2017. URL: http://arxiv.org/abs/1711.04964 . arXiv:1711.04964 . ↗
  87. H. Puerto, G. G. Sahin, I. Gurevych, Metaqa: Combining expert agents for multi-skill question answering, in: arXiv, volume abs/2112.01922, 2021. URL: https://arxiv.org/abs/2112. 01922 . arXiv:2112.01922 . — [arXiv:2112.01922](https://arxiv.org/abs/2112.01922) ↗
  88. A. Raganato, J. Camacho-Collados, R. Navigli, Word sense disambiguation: A unified evaluation framework and empirical comparison, in: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, Association for Computational Ling… find our card
  89. E. Barba, L. Procopio, R. Navigli, Consec: Word sense disambiguation as continuous sense comprehension, in: M. Moens, X. Huang, L. Specia, S. W. Yih (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Repub… find
  90. H. J. Levesque, E. Davis, L. Morgenstern, The Winograd Schema Challenge, in: Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’12, AAAI Press, Rome, Italy, 2012, pp. 552–561. URL: https://cs.nyu.edu/faculty/davise/papers/WSKR2012.pdf… ↗
  91. P. Edmonds, S. Cotton, SENSEVAL-2: Overview, in: Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems, Association for Computational Linguistics, Toulouse, France, 2001, pp. 1–5. URL: https://aclanthology.org/S01-1001 . ↗
  92. B. Snyder, M. Palmer, The English all-words task, in: Proceedings of SENSEVAL-3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, Association for Computational Linguistics, Barcelona, Spain, 2004, pp. 41–43. URL: https://aclanthology.org/W04-0811 . ↗
  93. S. Pradhan, E. Loper, D. Dligach, M. Palmer, SemEval-2007 task-17: English lexical sample, SRL and all words, in: Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), Association for Computational Linguistics, Prague, Czech Republic, 2007, pp. 87–92. URL: https://… find our card
  94. R. Navigli, D. Jurgens, D. Vannella, SemEval-2013 task 12: Multilingual word sense disambiguation, in: Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), Association for Computation… find
  95. A. Moro, R. Navigli, SemEval-2015 task 13: Multilingual all-words sense disambiguation and entity linking, in: Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), Association for Computational Linguistics, Denver, Colorado, 2015, pp. 288–297. URL: https://aclantholog… find
  96. C. FELLBAUM, Wordnet: An electronic lexical database, Computational Linguistics (1998) 292–296. find
  97. J. Kocoń, M. Gruza, J. Bielaniewicz, D. Grimling, K. Kanclerz, P. Miłkowski, P. Kazienko, Learning personal human biases and representations for subjective tasks in natural language processing, in: 2021 IEEE International Conference on Data Mining (ICDM), IEEE, 2021, pp. 1168–1173. doi: 10.1109/ICDM… — [doi:10.1109/icdm51629.2021.00140](https://doi.org/10.1109/icdm51629.2021.00140) find
  98. J. Bielaniewicz, K. Kanclerz, P. Miłkowski, M. Gruza, K. Karanowski, P. Kazienko, J. Kocoń, Deep-sheep: Sense of humor extraction from embeddings in the personalized context, in: 2022 IEEE International Conference on Data Mining Workshops (ICDMW), IEEE, 2022, pp. 967–974. find
  99. K. Kanclerz, M. Gruza, K. Karanowski, J. Bielaniewicz, P. Milkowski, J. Kocon, P. Kazienko, What if ground truth is subjective? personalized deep neural hate speech detection, in: Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022, European Language Resources Association, M… find
  100. P. Miłkowski, S. Saganowski, M. Gruza, P. Kazienko, M. Piasecki, J. Kocoń, Multitask personalized recognition of emotions evoked by textual content, in: 2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops), IEEE, 2022,… find
  101. P. Milkowski, M. Gruza, K. Kanclerz, P. Kazienko, D. Grimling, J. Kocon, Personal bias in prediction of emotions elicited by textual opinions, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P… find
  102. T. Gao, A. Fisch, D. Chen, Making pre-trained language models better few-shot learners, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Comp… find our card
  103. J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, D. C. Schmidt, A prompt pattern catalog to enhance prompt engineering with chatgpt, 2023. arXiv:2302.11382 . find our card
  104. T. Hagendorff, The ethics of AI ethics: An evaluation of guidelines, in: Minds Mach., volume 30, 2020, pp. 99–120. URL: https://doi.org/10.1007/s11023-020-09517-8 . doi: 10.1007/s11023-020-09517-8 . — [doi:10.1007/s11023-020-09517-8](https://doi.org/10.1007/s11023-020-09517-8) ↗
  105. A. Borji, A categorical archive of ChatGPT failures, in: arXiv, volume abs/2302.03494, 2023. URL: https://doi.org/10.48550/arXiv. 2302.03494 . doi: 10.48550/arXiv.2302.03494 . arXiv:2302.03494 . ↗
  106. W. Korczyński, J. Kocoń, Compression methods for transformers in multidomain sentiment analysis, in: 2022 IEEE International Conference on Data Mining Workshops (ICDMW), IEEE, 2022, pp. 419–426. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kocon-2023-chatgpt-jack-all-trades-master/