The ability for policies to generalize to new environments is key to the broadapplication of RL agents. A promising approach to prevent an agent’s policyfrom overfitting to a limited set of training environments is to apply regularizationtechniques originally developed for supervised learning. However, there are starkdifferences between supervised learning and RL. We discuss those differencesand propose modifications to existing regularization techniques in order to betteradapt them to RL. In particular, we focus on regularization techniques relying onthe injection of noise into the learned function, a family that includes some ofthe most widely used approaches such as Dropout and Batch Normalization. Toadapt them to RL, we proposeSelective Noise Injection(SNI), which maintainsthe regularizing effect the injected noise has, while mitigating the adverse effectsit has on the gradient quality. Furthermore, we demonstrate that the InformationBottleneck (IB) is a particularly well suited regularization technique for RL asit is effective in the low-data regime encountered early on in training RL agents.Combining the IB with SNI, we significantly outperform current state of the artresults, including on the recently proposed generalization benchmarkCoinrun.

Top