Utilizing a Multimodal Deep Learning Model to Identify Parkinson’s Disease from Voice Samples

Abstract

Dysarthria, a motor speech disorder affecting control of speech muscles, is a common early symptom of Parkinson’s Disease (PD), making voice analysis a promising tool for early detection. Acoustic biomarkers derived from sustained vowel phonations have shown potential for PD detection. This study develops a multimodal transformer model that integrates engineered acoustic features with log-Mel spectrogram embeddings to classify PD from sustained /a/ phonations. Eight acoustic features—mean fundamental frequency (F0), local jitter, local shimmer, detrended fluctuation analysis (DFA) exponent, pitch period entropy (PPE), recurrence period density entropy (RPDE), pitch variability, and harmonics-to-noise ratio (HNR)—were extracted using Librosa and Parselmouth and processed through a numerical branch. In parallel, log-Mel spectrograms were encoded with pretrained CNN backbones (ResNet18, EfficientNet_B0, MobileNet_V3_Large) in an image branch. A transformer layer integrated both modalities, with a final classifier predicting PD vs. healthy controls. Two datasets were combined: D1 with 81 recordings (41 HC, 40 PD; average ages 47.7 ± 14.3 and 67.0 ± 9.0 years) and D2 with 99 recordings (44 HC, 55 PD; average ages 67.1 ± 5.2 and 67.2 ± 8.7 years), yielding 180 recordings (85 HC, 95 PD). Data were split 80% training, 10% validation, and 10% testing. Models were trained with a learning rate of 1e-4, batch size 32, and 10 epochs across 5 independent runs. The best model achieved an accuracy of 0.93 ± 0.07, precision of 0.96 ± 0.05, recall of 0.91 ± 0.08, and F1-score of 0.94 ± 0.06, with stable training and validation loss convergence. These findings suggest that lightweight multimodal fusion of engineered acoustic and spectrographic features outperforms single-modality baselines and holds promise for scalable, noninvasive voice-based PD screening.