AdaDeltaUpdater Class Reference

Detailed Description

The class implements the AdaDelta method.

\[ \begin{array}{l} g_\theta=(1-\lambda){(\frac{ \partial f(\cdot) }{\partial \theta })}^2+\lambda g_\theta\\ d_\theta=\alpha\frac{\sqrt{s_\theta+\epsilon}}{\sqrt{g_\theta+\epsilon}}\frac{ \partial f(\cdot) }{\partial \theta }\\ s_\theta=(1-\lambda){(d_\theta)}^2+\lambda s_\theta \end{array} \]

.

where \( \frac{ \partial f(\cdot) }{\partial \theta } \) is a negative descend direction (eg, gradient) wrt \(\theta\), \(\lambda\) is a decay factor, \(\epsilon\) is used to avoid dividing by 0, \( \alpha \) is a build-in learning rate \(d_\theta\) is a corrected negative descend direction.

Reference: Matthew D. Zeiler, ADADELTA: An Adaptive Learning Rate Method, arXiv:1212.5701

Definition at line 58 of file AdaDeltaUpdater.h.

Inheritance diagram for AdaDeltaUpdater:

Public Member Functions
	AdaDeltaUpdater ()
	AdaDeltaUpdater (float64_t learning_rate, float64_t epsilon, float64_t decay_factor)
virtual	~AdaDeltaUpdater ()
virtual void	set_learning_rate (float64_t learning_rate)
virtual void	set_epsilon (float64_t epsilon)
virtual void	set_decay_factor (float64_t decay_factor)
virtual void	update_context (CMinimizerContext *context)
virtual void	load_from_context (CMinimizerContext *context)
virtual void	update_variable (SGVector< float64_t > variable_reference, SGVector< float64_t > raw_negative_descend_direction, float64_t learning_rate)
virtual void	set_descend_correction (DescendCorrection *correction)
virtual bool	enables_descend_correction ()

Protected Member Functions
virtual float64_t	get_negative_descend_direction (float64_t variable, float64_t gradient, index_t idx, float64_t learning_rate)

Protected Attributes
float64_t	m_build_in_learning_rate
float64_t	m_epsilon
float64_t	m_decay_factor
SGVector< float64_t >	m_gradient_accuracy
SGVector< float64_t >	m_gradient_delta_accuracy
DescendCorrection *	m_correction

Constructor & Destructor Documentation

AdaDeltaUpdater

(

)

Definition at line 36 of file AdaDeltaUpdater.cpp.

AdaDeltaUpdater	(	float64_t	learning_rate,
		float64_t	epsilon,
		float64_t	decay_factor
	)

Parameterized Constructor

Parameters

learning_rate	learning_rate
epsilon	epsilon
decay_factor	decay_factor

Definition at line 42 of file AdaDeltaUpdater.cpp.

~AdaDeltaUpdater ( )

virtual

Definition at line 73 of file AdaDeltaUpdater.cpp.

Member Function Documentation

virtual bool enables_descend_correction ( )

virtualinherited

Do we enable descend correction?

Returns

whether we enable descend correction

Definition at line 145 of file DescendUpdaterWithCorrection.h.

float64_t get_negative_descend_direction ( float64_t  variable, float64_t  gradient, index_t  idx, float64_t  learning_rate  )

protectedvirtual

Get the negative descend direction given current variable and gradient

It will be called at update_variable()

Parameters

variable	current variable (eg, \(\theta\))
gradient	current gradient (eg, \( \frac{ \partial f(\cdot) }{\partial \theta }\))
idx	the index of the variable
learning_rate	learning rate (for AdaDelta, learning_rate is NOT used because there is a build-in learning_rate)

Returns

negative descend direction (that is, \( d_\theta \))

Implements DescendUpdaterWithCorrection.

Definition at line 124 of file AdaDeltaUpdater.cpp.

void load_from_context ( CMinimizerContext *  context )

virtual

Return a context object which stores mutable variables Usually it is used in serialization.

This method will be called by FirstOrderMinimizer::load_from_context(CMinimizerContext* context)

Returns

a context object

Reimplemented from DescendUpdaterWithCorrection.

Definition at line 106 of file AdaDeltaUpdater.cpp.

void set_decay_factor ( float64_t  decay_factor )

virtual

Set decay_factor

Parameters

decay_factor

decay factor

Definition at line 65 of file AdaDeltaUpdater.cpp.

virtual void set_descend_correction ( DescendCorrection *  correction )

virtualinherited

Set the type of descend correction

Parameters

correction

the type of descend correction

Definition at line 135 of file DescendUpdaterWithCorrection.h.

void set_epsilon ( float64_t  epsilon )

virtual

Set epsilon

Parameters

epsilon

epsilon

Definition at line 58 of file AdaDeltaUpdater.cpp.

void set_learning_rate ( float64_t  learning_rate )

virtual

Set learning rate

Parameters

learning_rate

learning rate

Definition at line 51 of file AdaDeltaUpdater.cpp.

void update_context ( CMinimizerContext *  context )

virtual

Update a context object to store mutable variables

This method will be called by FirstOrderMinimizer::save_to_context()

Parameters

context

a context object

Reimplemented from DescendUpdaterWithCorrection.

Definition at line 86 of file AdaDeltaUpdater.cpp.

void update_variable ( SGVector< float64_t >  variable_reference, SGVector< float64_t >  raw_negative_descend_direction, float64_t  learning_rate  )

virtual

Update the target variable based on the given negative descend direction

Note that this method will update the target variable in place. This method will be called by FirstOrderMinimizer::minimize()

Parameters

variable_reference	a reference of the target variable
raw_negative_descend_direction	the negative descend direction given the current value
learning_rate	learning rate

Reimplemented from DescendUpdaterWithCorrection.

Definition at line 140 of file AdaDeltaUpdater.cpp.

Member Data Documentation

float64_t m_build_in_learning_rate

protected

learning_rate \( \alpha \) at iteration

Definition at line 141 of file AdaDeltaUpdater.h.

DescendCorrection* m_correction

protectedinherited

descend correction object

Definition at line 165 of file DescendUpdaterWithCorrection.h.

float64_t m_decay_factor

protected

decay term ( \( \lambda \))

Definition at line 147 of file AdaDeltaUpdater.h.

float64_t m_epsilon

protected

\( \epsilon \)

Definition at line 144 of file AdaDeltaUpdater.h.

SGVector<float64_t> m_gradient_accuracy

protected

\( g_\theta \)

Definition at line 150 of file AdaDeltaUpdater.h.

SGVector<float64_t> m_gradient_delta_accuracy

protected

\( s_\theta \)

Definition at line 153 of file AdaDeltaUpdater.h.

---

The documentation for this class was generated from the following files:

• AdaDeltaUpdater.h
• AdaDeltaUpdater.cpp