SGD¶

class dragon.vm.tensorflow.keras.optimizers.SGD(
learning_rate=0.01,
momentum=0.0,
nesterov=False,
name=None,
**kwargs
)[source]¶

The optimizer to apply SGD algorithm.

Following SGD algorithms are supported:

VanillaSGD, whose update is defined as:

\[\text{VanillaSGD}(g) = -\text{lr} * g \]

MomentumSGD [Polyak, 1964], whose update is defined as:

\[\text{MomentumSGD}(g) = \text{lr} * m_{t} \\ \quad \\ \text{where} \quad m_{t} = \text{momentum} * m_{t-1} + g \]

NesterovSGD [Sutskever et.al, 2013], whose update is defined as:

\[\text{NesterovSGD}(g) = \text{lr} * (\text{momentum} * m_{t} + g) \\ \quad \\ \text{where} \quad m_{t} = \text{momentum} * m_{t-1} + g \]

You can use one of them by setting the defaults:

# Set the ``lr`` only
vanilla_sgd = tf.keras.optimizers.SGD(learning_rate=0.1)

# Set the ``lr`` and ``momentum``
momentum_sgd = tf.keras.optimizers.SGD(learning_rate=0.1, momentum=0.9)

# Set the ``lr``, ``momentum`` and ``nesterov``
nesterov_sgd = tf.keras.optimizers.SGD(lr=0.1, momentum=0.9, nesterov=True)

init¶

SGD.__init__(
learning_rate=0.01,
momentum=0.0,
nesterov=False,
name=None,
**kwargs
)[source]¶

Create a SGD optimizer.

Parameters:

learning_rate (float, optional, default=0.01) – The initial value to \(\text{lr}\).
momentum (float, optional, default=0) – The initial value to \(\text{momentum}\).
nesterov (bool, optional, default=False) – True to switch to NesterovSGD optimizer.
name (str, optional) – The optional optimizer name.

Properties¶

iterations¶

Optimizer.iterations

Return the number of steps has run.

Returns:: int – The iterations.

Methods¶

apply_gradients¶

Optimizer.apply_gradients(grads_and_vars)[source]

Apply the gradients to update variables.