Adam¶

class dragon.optimizers.Adam(
  base_lr=0.001,
  beta1=0.9,
  beta2=0.999,
  eps=1e-08,
  **kwargs
)[source]¶

The optimizer to apply Adam algorithm. [Kingma & Ba, 2014].

The Adam update is defined as:

\[\text{Adam}(g) = -\frac{\text{lr} * m_{t}}{\sqrt{v_{t}} + \epsilon} \\ \quad \\ \text{where}\quad \begin{cases} m_{t} = \beta_{1} * m_{t-1} + (1 - \beta_{1}) * g \\ v_{t} = \beta_{2} * v_{t-1} + (1 - \beta_{2}) * g^{2} \end{cases} \]