Python v0.3.0a0+fdf26ef

RMSprop

class dragon.optimizers.RMSprop(
  base_lr=0.01,
  momentum=0.0,
  decay=0.9,
  eps=1e-08,
  **kwargs
)[source]

The optimizer to apply RMSprop algorithm. [Hinton et.al, 2013].

The RMSprop update is defined as:

\[\text{RMSprop}(g) = -m_{t} \\ \quad \\ \text{where} \quad \begin{cases} v_{t} = \text{decay} * v_{t-1} + (1 - \text{decay}) * g^{2} \\ m_{t} = \text{momentum} * m_{t-1} + \frac{\text{lr} * g}{\sqrt{v_{t}} + \epsilon} \end{cases} \]

__init__

RMSprop.__init__(
  base_lr=0.01,
  momentum=0.0,
  decay=0.9,
  eps=1e-08,
  **kwargs
)[source]

Create a RMSProp optimizer.

Parameters:
  • base_lr (float, optional, default=0.01) – The initial value for \(\text{lr}\).
  • momentum (float, optional, default=0.) – The initial value for \(\text{momentum}\).
  • decay (float, optional, default=0.9) – The initial value for \(\text{decay}\).
  • eps (float, optional, default=1e-8) – The initial value for \(\epsilon\).

Methods

apply_gradients

Optimizer.apply_gradients(
  values_and_grads,
  lr_mult=None,
  decay_mult=None
)[source]

Apply the gradients on values.

Parameters:
  • values_and_grads (Sequence[Sequence[dragon.Tensor]]) – The values and grads.
  • lr_mult (number, optional) – The multiplier to learning rate.
  • decay_mult (number, optional) – The multiplier to weight decay.