Python v0.3.0.dev+02ad90d
  • Overview

    • RMSprop

    class dragon.vm.torch.optim.RMSprop(
      params,
      lr=0.01,
      alpha=0.99,
      eps=1e-08,
      weight_decay=0,
      momentum=0,
      centered=False,
      scale=1,
      clip_norm=0
    )[source]

    The optimizer which implements RMSprop algorithm. [Hinton et.al, 2013].

    The RMSprop update is defined as:

    \[\text{RMSprop}(g) = -m_{t} \\ \quad \\ \text{where}\quad \begin{cases} v_{t} = \alpha * v_{t-1} + (1 - \alpha) * g^{2} \\ m_{t} = \text{momentum} * m_{t-1} + \frac{\text{lr} * g}{\sqrt{v_{t}} + \epsilon} \end{cases} \]

    __init__

    RMSprop.__init__(
      params,
      lr=0.01,
      alpha=0.99,
      eps=1e-08,
      weight_decay=0,
      momentum=0,
      centered=False,
      scale=1,
      clip_norm=0
    )[source]

    Create a RMSprop optimizer.

    Parameters:
    • params (Sequence[dragon.vm.torch.nn.Parameter]) – The parameters to optimize.
    • lr (float, required) – The initial value for \(\text{lr}\).
    • alpha (float, optional, default=0.99) – The initial value for \(\alpha\).
    • eps (float, optional, default=1e-7) – The initial value for \(\epsilon\).
    • weight_decay (float, optional, default=0) – The L2 penalty factor to weight.
    • momentum (float, optional, default=0) – The initial value for \(\text{momentum}\).
    • scale (float, optional, default=1) – The scaling factor to gradient.
    • clip_norm (float, optional, default=0) – The maximum L2 norm to clip gradient.

    Methods

    accumulate

    Optimizer.accumulate(momentum)[source]

    Accumulate the gradient of params.

    Call this method after each backward pass:

    x = torch.ones(1, requires_grad=True)
optimizer = torch.optim.SGD([x], lr=0.1)
for epoch in range(2):
    for step in range(3):
        y = x + 1
        y.backward()
        # Note to zero the accumulation at the first step
        optimizer.accumulate(momentum=1 if step > 0 else 1)
    optimizer.step()
print(x)  # 0.4
    Parameters:
    • momentum (float, required) – The momentum to the accumulated value.

    add_param_group

    Optimizer.add_param_group(param_group)[source]

    Add a new param group into the optimizer.

    The param_group should be a dict containing the defaults optionally:

    # A group redefined ``lr`` and ``weight_decay``
param_group1 = {
    'params': [],
    'lr': 0.01,
    'weight_decay': 0.0001,
}
# A group inherits the defaults while using ``multiplier``
param_group2 = {
    'params': [],
    'lr_mult': 1,
    'decay_mult': 1,
}
    Parameters:
    • param_group (dict) – The param group to add.

    step

    Optimizer.step()[source]

    Perform one step update.

    Call this method after a backward pass:

    x = torch.ones(1, 3, requires_grad=True)
y = x + 1
y.backward()
optimizer.step()

    zero_grad

    Optimizer.zero_grad(reset=False)[source]

    Set the gradient of params to zero.

    This method is not necessary usually, as we will overwrite the gradients in the next computation.

    However, if some gradients are not computed every time, remember to reset them before step(...):

    m1 = torch.nn.Linear(3, 3)
m2 = torch.nn.Linear(3, 3)
x = torch.ones(1, 3, requires_grad=True)
for i in range(10):
    x = m1(x)
    if i in (2, 4, 6):
        x += m2(x)
optimizer.zero_grad(reset=True)
x.backward()
optimizer.step()
    Parameters:
    • reset (bool, optional, default=False) – True to reset the memory instead of zeroing.