loading page

Is it possible for ChatGPT to mimic human annotator?
  • +3
  • Oliwier Kaszyca,
  • Przemysław Kazienko,
  • Jan Kocoń,
  • Igor Cichecki,
  • Mateusz Kochanek,
  • Dominka Szydło
Oliwier Kaszyca

Corresponding Author:[email protected]

Author Profile
Przemysław Kazienko
Jan Kocoń
Igor Cichecki
Mateusz Kochanek
Dominka Szydło

Abstract

The development of large language models, such as ChatGPT (GPT-3.5) and GPT-4, has revolutionized natural language processing (NLP) and opened up new possibilities in various fields. These models demonstrate remarkable capabilities in generating coherent and contextually relevant text, making them suitable for a wide range of applications. This work focuses on automatic text annotation in subjective problems and person-alization using ChatGPT. The primary objective is to investigate the ChatGPT generative capabilities and evaluate its performance in classification and regression NLP tasks. Furthermore, the work also contributes a novel methodology for evaluating personalized ChatGPT and adapting it to address specific problem domains. The results obtained from multiple experimental setups showcase the potential of the method to automatically exploit ChatGPT to generate text annotations. However, the conclusions drawn from the research highlight the need for a further detailed and more extensive analysis across multiple problem domains and diverse datasets.
12 Dec 2023Submitted to TechRxiv
14 Dec 2023Published in TechRxiv