public abstract class AnnealingSchedule extends Object
AnnealingSchedule
instance implements a method to
return the learning rate for a specified epoch. It also has a
method to receive callback reports on the resulting error rate for
an epoch so that implementations may adapt the learning rate to
received error.
This class provides an abstract base class implementation along with static factory methods to create the two most popular annealing schedules, constant, exponential decay and inverse scaling.
The contant learning rate schedule always returns the same learning rate, which is fixed at construction time.
The exponential decay annealing schedule sets the learning rate as follows based on a specfieid exponential base:
Under this schedule, the learning rate undergoes exponential decay starting at the initial learning rate and decaying exponentially at a rate determined by the base of the exponent.learningRate(epoch) = initialLearningRate * java.lang.Math.pow(base,epoch)
The exponential learning rate can find solutions quickly in fairly wellbehaved spaces, but may stop short of the minimum error solution due to too much decay in later epochs.
The inverse learning rate scaling sets the learning rate as:
The inverse scaling annealing schedule lowers the rate more quickly than the exponential rates initially and then more slowly for later epochs.learningRate(epoch) = initialLearningRate / (1 + epoch/annealingRate)
This is a popular learning rate because it is guaranteed to converge in the limit. It can be slower to converge once it gets near a solution than exponential decay.
receivedError()
method is a
boolean flag indicating whether to accept the updates to the
underlying vectors or not. This allows samplingbased annealing
schedules to be implemented that evaluate several learning
rates and accept just the one with the most error reduction.
The method allowsRejection()
should be overridden
to return false
if the annealing schedule never rejects
updates; this will save a coefficient vector copy per epoch
in logistic regression.
The inverse scale metric is popular because it is theoretically guaranteed to converge in the limit.
An annealing rate will converge in the limit within arbitrary precision of a solution if the learning rate satisfies:
and:Σ_{epoch} learningRate(epoch) = ∞
Σ_{epoch} learningRate(epoch)^{2} < ∞
Constructor and Description 

AnnealingSchedule()
Donothing constructor to be used by concrete implementations.

Modifier and Type  Method and Description 

boolean 
allowsRejection()
Return
true if this annealing schedule allows
updates to be rejected. 
static AnnealingSchedule 
constant(double learningRate)
Return the annealing schedule for the specified constant learning
rate.

static AnnealingSchedule 
exponential(double initialLearningRate,
double base)
Return the exponential annealing schedule with the specified
initial learning rate and exponent.

static AnnealingSchedule 
inverse(double initialLearningRate,
double annealingRate)
Return the inverse annealing schedule with the specified
initial learning rate and annealing rate.

abstract double 
learningRate(int epoch)
Return the learning rate for the specified epoch.

boolean 
receivedError(int epoch,
double rate,
double error)
Receive a report from an optimizer about the effect of the
specified learning rate in the specified epoch and return
true if the update producing the error should
be accepted or rejected. 
public AnnealingSchedule()
public boolean allowsRejection()
true
if this annealing schedule allows
updates to be rejected. The implementation in this
class always returns true
for backward compatibility
reasons. Implementations that never reject updates
should override this method to return false
.true
if this schedule allows update rejections.public abstract double learningRate(int epoch)
epoch
 Epoch being evaluated.public boolean receivedError(int epoch, double rate, double error)
true
if the update producing the error should
be accepted or rejected.
This abstract class's implementation of this method is to do nothing. Concrete subclasses which adapt learning rates based on empirical error reports from the optimizer must override this method.
epoch
 Training epoch.rate
 Training rate.error
 Training error.public static AnnealingSchedule inverse(double initialLearningRate, double annealingRate)
initialLearningRate
 Initial learning rate for epoch zero.annealingRate
 Rate at which initial learning rate anneals.IllegalArgumentException
 If the initial learning rate or
the annealing rates are not finite and positive.public static AnnealingSchedule exponential(double initialLearningRate, double base)
initialLearningRate
 Initial learning rate for epoch 0.base
 Base of the exponential decay.IllegalArgumentException
 If the initial learning rate is
not finite and positive, or if the exponent is not between 0.0 (exclusive)
and 1.0 (inclusive).public static AnnealingSchedule constant(double learningRate)
learningRate
 The constant rate returned by this
annealing schedule.IllegalArgumentException
 If the learning rate is not
finite and positive.