public abstract class AnnealingSchedule extends Object
AnnealingScheduleinstance implements a method to return the learning rate for a specified epoch. It also has a method to receive callback reports on the resulting error rate for an epoch so that implementations may adapt the learning rate to received error.
This class provides an abstract base class implementation along with static factory methods to create the two most popular annealing schedules, constant, exponential decay and inverse scaling.
The contant learning rate schedule always returns the same learning rate, which is fixed at construction time.
The exponential decay annealing schedule sets the learning rate as follows based on a specfieid exponential base:
Under this schedule, the learning rate undergoes exponential decay starting at the initial learning rate and decaying exponentially at a rate determined by the base of the exponent.learningRate(epoch) = initialLearningRate * java.lang.Math.pow(base,epoch)
The exponential learning rate can find solutions quickly in fairly well-behaved spaces, but may stop short of the minimum error solution due to too much decay in later epochs.
The inverse learning rate scaling sets the learning rate as:
The inverse scaling annealing schedule lowers the rate more quickly than the exponential rates initially and then more slowly for later epochs.learningRate(epoch) = initialLearningRate / (1 + epoch/annealingRate)
This is a popular learning rate because it is guaranteed to converge in the limit. It can be slower to converge once it gets near a solution than exponential decay.
receivedError()method is a boolean flag indicating whether to accept the updates to the underlying vectors or not. This allows sampling-based annealing schedules to be implemented that evaluate several learning rates and accept just the one with the most error reduction.
allowsRejection() should be overridden
false if the annealing schedule never rejects
updates; this will save a coefficient vector copy per epoch
in logistic regression.
The inverse scale metric is popular because it is theoretically guaranteed to converge in the limit.
An annealing rate will converge in the limit within arbitrary precision of a solution if the learning rate satisfies:
and:Σepoch learningRate(epoch) = ∞
Σepoch learningRate(epoch)2 < ∞
|Constructor and Description|
Do-nothing constructor to be used by concrete implementations.
|Modifier and Type||Method and Description|
Return the annealing schedule for the specified constant learning rate.
Return the exponential annealing schedule with the specified initial learning rate and exponent.
Return the inverse annealing schedule with the specified initial learning rate and annealing rate.
Return the learning rate for the specified epoch.
Receive a report from an optimizer about the effect of the specified learning rate in the specified epoch and return
public boolean allowsRejection()
trueif this annealing schedule allows updates to be rejected. The implementation in this class always returns
truefor backward compatibility reasons. Implementations that never reject updates should override this method to return
trueif this schedule allows update rejections.
public abstract double learningRate(int epoch)
epoch- Epoch being evaluated.
public boolean receivedError(int epoch, double rate, double error)
trueif the update producing the error should be accepted or rejected.
This abstract class's implementation of this method is to do nothing. Concrete subclasses which adapt learning rates based on empirical error reports from the optimizer must override this method.
epoch- Training epoch.
rate- Training rate.
error- Training error.
public static AnnealingSchedule inverse(double initialLearningRate, double annealingRate)
initialLearningRate- Initial learning rate for epoch zero.
annealingRate- Rate at which initial learning rate anneals.
IllegalArgumentException- If the initial learning rate or the annealing rates are not finite and positive.
public static AnnealingSchedule exponential(double initialLearningRate, double base)
initialLearningRate- Initial learning rate for epoch 0.
base- Base of the exponential decay.
IllegalArgumentException- If the initial learning rate is not finite and positive, or if the exponent is not between 0.0 (exclusive) and 1.0 (inclusive).
public static AnnealingSchedule constant(double learningRate)
learningRate- The constant rate returned by this annealing schedule.
IllegalArgumentException- If the learning rate is not finite and positive.