Exploring the Use of Synthetic Data from AI Chatbots for Predicting Alzheimer's Disease: Methods for Validation and Barriers to Real-World Implementation

Abstract

Artificial intelligence has rapidly advanced in the last decade for applications in healthcare from diagnostic tools to predictive analytics. Machine learning, a subset of the wide range of AI tools undergoing implementation, uses its sophisticated abilities to recognize patterns in subsequent data sets to help enable the early detection of chronic illnesses like Alzheimer's disease. Atenacious challenge persists in the reliance of real world data, when developing machine learning models, raising ethical concerns over privacy, security, and bias. Synthetic data offers a potential solution by mimicking real-world medical datasets while avoiding privacy issues.  Traditional approaches rely on statistical analytics and simulations to generate viable data sets, unlike this research, which will be proposing a new source of data generation: ChatGPT.  Exploring AI-generated synthetic data shows promise for a future in enhanced early diagnostic capabilities, and if successful, goes far beyond Alzheimer's.

A dataset from the University of Southern California’s Image and Data Archive was selected as a trusted reference source and used as the foundation for model testing. Key variables, particularly age distribution and cognitive classification groups (CN, EMCI, LMCI, and MCI), were analyzed to identify patterns associated with Alzheimer’s disease progression. These observed patterns were then used to guide the generation of synthetic data via a chatbot, ensuring that the synthetic dataset reflected realistic demographic distributions and cognitive stage relationships for training the machine learning model.

The machine learning model was trained on the synthetic dataset to predict Alzheimer’s diagnosis based on symptom-related variables. The model achieved an accuracy of 1.00, with perfect precision, recall, and F1-scores, indicating no misclassifications. While impressive, these results suggest overfitting due to limited data diversity and dominant features such as MMSE score and hippocampal volume. Additionally, the inclusion of Patient_ID as a feature likely contributed to memorization rather than generalizable learning.