Examining the Human-Like Proficiency of GPT-2 in Recognizing Self-Generated Texts
DOI:
https://doi.org/10.58445/rars.768Keywords:
natural language processing, gpt, LLMAbstract
In recent years, the widespread use of generative language models has brought opportunities as well as some philosophical and technical questions. GPT-2, a language model with 1.5B parameters, is an open-source language model provided by OpenAI. Our aim in this paper is to utilize the classification capabilities of GPT-2 to create a new perspective on the question of whether language models show some kind of consciousness/self-awareness, in addition to technical questions such as how to detect the misuse of the outputs of language models.
To investigate this phenomenon, GPT-2ForSequenceClassification model was fine-tuned on TuringBench datasets and its performance was examined. In addition, the accuracy achieved by model as a result of training with training sets of different sizes, as well as its performance in human-machine discrimination, were evaluated.
The model exhibits consistent and above-average performance in identifying GPT-2-generated content compared to its classification accuracy in distinguishing other machine-generated text from human writing. This performance of the model in understanding self-generated texts is very similar to people's ability to recognize their own writing, and these results offer an interesting perspective on the self-awareness of artificial intelligence. Additionally, the model showed high accuracy in distinguishing machine generated output from human output, even when trained with very few examples.
References
Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Scaling expert language models with unsupervised domain discovery, 2023.
Kevin P. Murphy. Machine learning - a probabilistic perspective. In Adaptive computation and machine learning series, 2012.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www. deeplearningbook.org.
Andreas Kamilaris and Francesc X Prenafeta-Boldú. Deep learning in agriculture: A survey. Computers and electronics in agriculture, 147:70–90, 2018.
Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(7):3523–3542, 2021.
Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean. A guide to deep learning in healthcare. Nature medicine, 25(1):24–29, 2019.
Erik Cambria and Bebo White. Jumping nlp curves: A review of natural language processing research [review article]. IEEE Computational Intelligence Magazine, 9(2):48–57, 2014.
Daniel W Otter, Julian R Medina, and Jugal K Kalita. A survey of the usages of deep learning for natural language processing. IEEE transactions on neural networks and learning systems, 32(2):604–624, 2020.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large language models, 2023.
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. AI Open, 3:111–132, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023.
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models, 2023.
Alec Radford, Jeff Wu, Rewon Child, D. Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
Partha Pratim Ray. Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems, 3:121–154, 2023.
Albert Gatt and Emiel Krahmer. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation, 2018.
Krystal Hu. Chatgpt sets record for fastest-growing user base - analyst note, 2023.
Steven Brown. The “who” system of the human brain: A system for social cognition about the self and others. Front. Hum. Neurosci., 14:224, June 2020.
Evan Crothers, Nathalie Japkowicz, and Herna Viktor. Machine generated text: A comprehensive survey of threat models and detection methods, 2023.
Downloads
Posted
Categories
License
Copyright (c) 2023 Ömer Can Kuşcu, Adem Mert Akkaya
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.