A-EYE IN THE SKY: Examining Machine Learning-Based Keystroke Reconstruction from Passive Video Capture

Abstract

The rapid growth of artificial intelligence (AI) has allowed for unprecedented progress across disciplines, but has also introduced new cybersecurity threats. Among these are machine-learning (ML) powered side-channel attacks. In this paper, we explore optical side-channel attacks, and the role ML plays in this attack vector. We investigate the feasibility of vision-based keystroke inference as an automated attack vector. Using a publicly available dataset of typing videos from Hugging Face, we frame the problem as a supervised classification task and systematically explore a minimal yet effective Neural Network (NN) pipeline. Our experiments compare five neural network architectures: EfficientNet_B0, ResNet-8, MobileNetV2_100, ConvNeXt_Tiny, along with a Feed-Forward multilayer perceptron (MLP). We also explore ten critical hyperparameters and discuss how variations affect performance. We train 26 models with varying hyperparameters, using Kaggle P100 GPUs. We evaluate models using F1 score, training loss, validation loss, and validation accuracy. Results indicate that a pretrained MobileNetV2-100 architecture, trained on min-max normalized, transformed and generously labeled data without time context, with class balanced training, a hidden layer, dropout regularization, and a time-based learning-rate schedule, achieves the best performance with a peak F1 score of 0.542. Our findings demonstrate that effective keystroke inference can be achieved with relatively modest computational resources and without sophisticated preprocessing or postprocessing. This suggests that such attacks are within reach of moderately skilled adversaries and highlights the need for countermeasures to such attack vectors. We discuss limitations of this work, chiefly dataset frame-label mismatch, and propose directions for future work such as blind keystroke recognition and inference on mobile devices.