Mediapipe Fingering Classification for Violin

Abstract

The extent of the potential of Mediapipe in analyzing and classifying complex hand patterns, like those found in violin hand movements, remains elusive. This research explores the feasibility of using Mediapipe, an open-source framework that allows the extraction of finger position key points from images/videos, to detect violin fingering positions for playing a virtual instrument. Further, we explore the limitations and potential of Mediapipe for recognizing complex hand movements involved in playing the violin by using a combination of Google Colab and support vector classification models. The repeated training on misclassified images was able to take the F1 score from an initial value of 0.82 to a final value of 0.95. This initial research shows the potential for a video based virtual violin instrument. Further areas of exploration could include expanding the total image training data and testing viability of other classifiers.