Generating Classical Music With Transformers

Abstract

Models based on the Transformer Architecture (Vaswani et. al. 2017) have been a staple in the landscape of autoregressive generation for nearly a decade as of the publishing of this paper. While such models have proven to be proficient in areas such as Text Generation in GPTs (Generative Pre-Trained Transformers) and NLP (Natural Language Processing), their potential for autoregressive generation is yet unproven in other similar areas, especially in the field of Music Generation. With the goal of exploring the potential of the transformer architecture in more novel areas, we propose a small, 57M-parameter model built on a standard Transformer Architecture and trained on classical piano music in the MIDI format. We train on a dataset that primarily contains music from the Classical and Romantic periods, with music from the more modern periods also included, albeit at a smaller scale. To help our model better understand the nuances of classical music that are absent in conventional text applications, we introduce a novel Musical Loss Function that will work in tandem with standard loss functions, specifically Cross-Entropy Loss, to ensure coherent and melodious generation. Through our model, we aim to prove that ~50M-parameter models trained on smaller, meticulously selected datasets can compete with larger music generation models trained on larger datasets in terms of performance.