|
SHOGUN
4.1.0
|
The class implements the AdaGrad method.
\[ \begin{array}{l} g_\theta={(\frac{ \partial f(\cdot) }{\partial \theta })}^2+g_\theta\\ d_\theta=\alpha\frac{1}{\sqrt{g_\theta+\epsilon}}\frac{ \partial f(\cdot) }{\partial \theta }\\ \end{array} \]
where \( \frac{ \partial f(\cdot) }{\partial \theta } \) is a negative descend direction (eg, gradient) wrt \(\theta\), \(\epsilon\) is used to avoid dividing by 0, \( \alpha \) is a build-in learning rate \(d_\theta\) is a corrected negative descend direction.
Duchi, John, Elad Hazan, and Yoram Singer. "Adaptive subgradient methods for online learning and stochastic optimization." The Journal of Machine Learning Research 12 (2011): 2121-2159.
在文件 AdaGradUpdater.h 第 56 行定义.
Public 成员函数 | |
| AdaGradUpdater () | |
| AdaGradUpdater (float64_t learning_rate, float64_t epsilon) | |
| virtual | ~AdaGradUpdater () |
| virtual void | set_learning_rate (float64_t learning_rate) |
| virtual void | set_epsilon (float64_t epsilon) |
| virtual void | update_context (CMinimizerContext *context) |
| virtual void | load_from_context (CMinimizerContext *context) |
| virtual void | update_variable (SGVector< float64_t > variable_reference, SGVector< float64_t > raw_negative_descend_direction, float64_t learning_rate) |
| virtual void | set_descend_correction (DescendCorrection *correction) |
| virtual bool | enables_descend_correction () |
Protected 成员函数 | |
| virtual float64_t | get_negative_descend_direction (float64_t variable, float64_t gradient, index_t idx, float64_t learning_rate) |
Protected 属性 | |
| float64_t | m_build_in_learning_rate |
| float64_t | m_epsilon |
| SGVector< float64_t > | m_gradient_accuracy |
| DescendCorrection * | m_correction |
| AdaGradUpdater | ( | ) |
在文件 AdaGradUpdater.cpp 第 36 行定义.
| AdaGradUpdater | ( | float64_t | learning_rate, |
| float64_t | epsilon | ||
| ) |
Parameterized Constructor
| learning_rate | learning_rate |
| epsilon | epsilon |
在文件 AdaGradUpdater.cpp 第 42 行定义.
|
virtual |
在文件 AdaGradUpdater.cpp 第 64 行定义.
|
virtualinherited |
Do we enable descend correction?
在文件 DescendUpdaterWithCorrection.h 第 145 行定义.
|
protectedvirtual |
Get the negative descend direction given current variable and gradient
It will be called at update_variable()
| variable | current variable (eg, \(\theta\)) |
| gradient | current gradient (eg, \( \frac{ \partial f(\cdot) }{\partial \theta }\)) |
| idx | the index of the variable |
| learning_rate | learning rate (for AdaGrad, learning_rate is NOT used because there is a build-in learning_rate) |
实现了 DescendUpdaterWithCorrection.
在文件 AdaGradUpdater.cpp 第 98 行定义.
|
virtual |
Return a context object which stores mutable variables Usually it is used in serialization.
This method will be called by FirstOrderMinimizer::load_from_context(CMinimizerContext* context)
重载 DescendUpdaterWithCorrection .
在文件 AdaGradUpdater.cpp 第 87 行定义.
|
virtualinherited |
Set the type of descend correction
| correction | the type of descend correction |
在文件 DescendUpdaterWithCorrection.h 第 135 行定义.
|
virtual |
|
virtual |
|
virtual |
Update a context object to store mutable variables
This method will be called by FirstOrderMinimizer::save_to_context()
| context | a context object |
重载 DescendUpdaterWithCorrection .
在文件 AdaGradUpdater.cpp 第 75 行定义.
|
virtual |
Update the target variable based on the given negative descend direction
Note that this method will update the target variable in place. This method will be called by FirstOrderMinimizer::minimize()
| variable_reference | a reference of the target variable |
| raw_negative_descend_direction | the negative descend direction given the current value |
| learning_rate | learning rate |
重载 DescendUpdaterWithCorrection .
在文件 AdaGradUpdater.cpp 第 108 行定义.
|
protected |
learning_rate \( \alpha \) at iteration
在文件 AdaGradUpdater.h 第 134 行定义.
|
protectedinherited |
descend correction object
在文件 DescendUpdaterWithCorrection.h 第 165 行定义.
|
protected |
\( epsilon \)
在文件 AdaGradUpdater.h 第 137 行定义.
\( g_\theta \)
在文件 AdaGradUpdater.h 第 140 行定义.
