AdaGrad, short for adaptive gradient, institutes a per-parameter learning rate rather than a globally-shared rate.

Optimizers with live results:

Stochastic Gradient Descent:

Optimizer: SGD. Learning Rate: 1.0.

Optimizer: SGD. Learning Rate: 0.5.

Optimizer: SGD. Learning Rate: 1.0. Decay: 1e-2.

Optimizer: SGD. Learning Rate: 1.0. Decay: 1e-3.

Optimizer: SGD. Learning Rate: 1.0. Decay: 1e-3. Momentum: 0.5.

Optimizer: SGD. Learning Rate: 1.0. Decay: 1e-3. Momentum: 0.9.

AdaGrad:

Optimizer: AdaGrad. Decay: 1e-4

RMSProp:

Optimizer: RMSProp. Decay: 1e-4

Optimizer: RMSProp. Decay: 1e-5. rho: 0.999

Adam:

Optimizer: Adam. Learning Rate: 0.02. Decay: 1e-5

Optimizer: Adam. Learning Rate: 0.05. Decay: 5e-7