Hyperparameter Optimization for Deep Reinforcement Learning

An Atari Breakout Case Study

Abstract

In reinforcement learning (RL), a subfield of machine learning, we train systems to perform complex tasks through trial and error. In RL, an agent interacts with an environment, taking actions that generate a cumulative reward. If a series of actions generates a high cumulative reward, those actions will be favorable in the future. Some applications of RL include improving the performance of self-driving cars, improving the performance of large language models, and playing games like Go. While playing games might not have a direct impact on the real world, systems like AlphaGo have helped improve our understanding of RL that aids in more real-world applications. This study uses game completion as a test bed to better understand the underlying mechanisms behind RL, specifically the effects of tuning hyperparameters on a model’s performance. A Deep Q-Learning (DQN) model architecture was chosen for this analysis, and we tuned batch size, learning rate, exploration rate, and discount factor. We hypothesized optimizing these hyperparameters would increase the cumulative reward. These hyperparameters were tuned to maximize the score in the game of Atari Breakout. We found that altering the discount factor to be greater than or less than one results in a much less effective model, whereas tuning hyperparameters that were changed caused little change to the performance. The results of this study can be used to improve the performance of future RL models. All code to reproduce results in this study is available at: https://github.com/BobyWoby/Reinforcement-Learning.